Commit Graph

33266 Commits

Author SHA1 Message Date
Anders Kaseorg d0634181b5 styles: Fix left sidebar indentation for PostCSS migration.
cssnano reduces this to a constant in a production build.  (We could
add postcss-calc if we wanted this reduced in development.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-30 15:06:29 -07:00
Mateusz Mandera d70e1bcdb7 settings: Add FAKE_EMAIL_DOMAIN setting.
Fixes #9401.

This adds a FAKE_EMAIL_DOMAIN setting, which should be used if
EXTERNAL_HOST is not a valid domain, and something else is needed to
form bot and dummy user emails (if email visibility is turned off).
It defaults to EXTERNAL_HOST.

get_fake_email_domain() should be used to get this value. It validates
that it's correctly set - that it can be used to form valid emails.

If it's not set correctly, an exception is raised. This is the right
approach, because it's undesirable to have the server seemingly
peacefully operating with that setting misconfigured, as that could
mask some hidden sneaky bugs due to UserProfiles with invalid emails,
which would blow up the moment some code that does validate the emails
is called.
2019-08-30 14:59:00 -07:00
Anders Kaseorg 4de3bbeafa styles: Finish removing manual antialiasing configuration.
Followup from commit ddb965110f.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-30 14:51:52 -07:00
Anders Kaseorg abbd8a7f45 styles: Remove most vendor-prefixed CSS attributes.
Many of them are now automatically generated by autoprefixer, while
others are unnecessary based on .browserslistrc, and some were just
wrong (the linear-gradient based checkerboard pattern in lightbox has
been broken in Firefox for a while).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-30 14:51:52 -07:00
Tim Abbott 62c9ea7cf9 linkifiers: Fix problems with capture groups called "name".
Apparently, due to poor naming of the outer capture group we use to
separate the actual match from the surrounding whitespace (etc.) we
use to determine if the syntax is a possible linkifier start/end, if
you created a linkifier using "name" as the capture group, we'd try to
compile a pattern with two capture groups called "name", which would
500, preventing anyone from accessing the organization.
2019-08-30 09:36:14 -07:00
Anders Kaseorg 096ef1445f parse_os_release: Use /etc/os-release always; remove DISTRIB_FAMILY.
To replace DISTRIB_FAMILY, there’s now an os_families function using
the standard ID and ID_LIKE information in /etc/os-release.

Fixes #13070; fixes #13071.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 17:30:20 -07:00
Anders Kaseorg 875002108f setup_venv: Remove CentOS workaround for fixed pycurl bug.
We are installing pycurl 7.43.0.3 which includes the fix.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:38:38 -07:00
Anders Kaseorg d312d04510 styles: Replace Sass with PostCSS.
It’s about as fast as node-sass (faster, according to their
benchmarks) and more flexible.  Autoprefixer is neat: we can now go
delete all our -moz-, -webkit-, etc. lines and have them autogenerated
as necessary based on .browserslistrc.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:35:51 -07:00
Anders Kaseorg e0d122217b provision: Use pre-built pgroonga on Debian 10.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:35:29 -07:00
Tim Abbott 73c9450d8a docs: Expand documentation on using IDs in narrows. 2019-08-29 16:17:09 -07:00
Tim Abbott 8ef610f8c8 provision: Remove now-duplicate extra Ubuntu section. 2019-08-29 12:53:10 -07:00
Anders Kaseorg db44d61aab setup-apt-repo: Remove PPA and packagecloud repository.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer).  So we no longer need to
provide/install a PPA at all.

Closes #13027.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:53:04 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
Anders Kaseorg c3e395b7d8 search: Reimplement ts_locs_array in pure PostgreSQL.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:59:12 -07:00
Anders Kaseorg d2d4800b1c test_narrow: Compile queries with the current SQLAlchemy dialect.
We’re about to start using PostgreSQL-specific syntax that can’t be
stringified without a specified dialect.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:58:01 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Tim Abbott 83a012eaeb lint: Improve over-aggressive pattern botserver->Botserver pattern.
This is still not a great pattern, but this is sufficient to avoid it
firing on URLs of the form zulipbotserver.example.com.
2019-08-28 16:18:16 -07:00
Anders Kaseorg 9e481e353a .yarnrc: Set ignore-scripts true.
Follow up to #13065, to keep manual yarn invocations consistent with
our automated ones.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 16:15:54 -07:00
Rohitt Vashishtha 6fe99171c3 docs/botserver: Add basic troubleshooting help.
It could be a non trivial task ensuring that the Zulip server can
access the Botserver. These steps examine some of those cases.
2019-08-28 16:08:34 -07:00
Rohitt Vashishtha 7f3ee1b6b1 docs: Fix typo: Heroko -> Heroku. 2019-08-28 15:59:03 -07:00
Rohitt Vashishtha 16227ed23a docs/botserver: Explain how the botserver works.
Understanding the basic model can help people troubleshoot any
errors easily.
2019-08-28 15:59:03 -07:00
Tim Abbott c06d8129a3 custom profile fields: Add placeholder for custom URL.
This makes the feature useable for someone who hasn't used it before.
2019-08-28 15:43:35 -07:00
Yashashvi Dave 9e343f1a68 custom fields: Update create-field UI for default external account fields. 2019-08-28 15:39:20 -07:00
Yashashvi Dave 313003900a custom fields: Move field-type inputs first in create-field form.
As other data of field, such as field name, hint etc. are
relative to field type, this commit moves the field type
input to the first order in create field form in org settings.
2019-08-28 15:39:13 -07:00
Yashashvi Dave e05429b6a9 custom fields: Add default external account custom fields. 2019-08-28 15:35:53 -07:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
David Rosa 4afdfa6055 docs: Place warning at the top of every page in /latest/production.
- Added warning block to dev docs using _templates/layout.html
- Removed copy-pasted warning from install.md and requirements.md
- Removed unreleased tag in docs/conf.py that's no longer used

Useful ref: https://www.sphinx-doc.org/en/master/templating.html

Fixes #13056.
2019-08-28 15:31:47 -07:00
Wyatt Hoodes e64b5a2b88 data export: Fix success banner not clearing.
There was a bug where the success banner stuck
around even after the export completed.  We now
nicely fade and remove the banner upon a successful
population of the export in the table.

Fixes: #13045
2019-08-28 15:23:39 -07:00
Wyatt Hoodes 2a020fa6cc data export: Fix sort feature regression.
02cfb47 removed a couple HTML tags that were
being used to sort the table.  We fix this,
but disable filtering exports by marking the
input type as `hidden`.  We use this approach as
it seems `list_render` doesn't like an
undefined `opts.filter.element`, which is
what happens if we simply remove the `filter`
key.
2019-08-28 15:23:39 -07:00
Vinit Singh d09a80260b lint: Replace local variables named 'msgid' with 'message_id'.
Follow up of commit 2a1305d. Replace all local variables named 'msgid'
with 'message_id' in all JS and HTML files, and adds a linter rule for
it as well.

Resolves #12952.
2019-08-28 15:19:30 -07:00
Greg Price c6fb8c048a docs: Add troubleshooting entry for TLS <1.2 servers.
This corresponds to a user report at zulip/zulip-mobile#3596 .
2019-08-27 16:25:37 -07:00
Mateusz Mandera 6bd34afa0d models: Migration of UserMessage.id to bigint, part 2.
This requires part 1 (which can take hours to run but generally
doesn't require downtime) to be completed first.

This portion of the migration will require the server to be completely
down for a brief period; for chat.zulip.org with 250M UserMessage
rows, it took about 60s to run; that time will vary depending on
hardware details like whether the server has an SSD, but fundamentally
shouldn't be long.

Our upgrade-zulip and upgrade-zulip-from-git tools can apply this
migration correctly; nothing special needs to be done.

Fixes #13040.
2019-08-26 21:25:19 -07:00
Mateusz Mandera b008515d63 models: Migration of UserMessage.id to bigint, part 1.
As part of adding support for more than 2B UserMessage rows in a Zulip
server, we need to change UserMessage.id (a field we don't access but
is needed by Django) from an int to a bigint.  This commit is a series
of migrations which create a `bigint_id` column and populates it correctly.

This migration will take a long time to run; on chat.zulip.org (a
server with a lot of history), it took about 4 hours to complete.

How to migrate with minimal downtime:

1. Run `upgrade-zulip-from-git` through this commit.  It will install
migration 0238 and then more or less hang while applying migration
0239.  Once migration 0238 is completed, however, your server should
be able to be started back up safely while migration 0239 is running.

2. Run `/home/zulip/deployments/next/scripts/restart-server` in a
separate terminal to get Zulip running again.

3. When the `upgrade-zulip-from-git` command finishes, it will
automatically re-restart the Zulip server, leaving you in a consistent
state and ready to do part 2 of the migration.

A useful `manage.py shell` query for checking the state after this
commit is consistent is this:

    assert UserMessage.objects.exclude(bigint_id=F("id")).count() == 0

Part of #13040.
2019-08-26 21:14:15 -07:00
Tim Abbott 89aeefed76 urls: Tighten many unintentionally broad URL patterns.
Previously, several of our URL patterns accidentally did not end with
`$`, and thus ended up controlling just the stated URL, but actually a
much broader set of URLs starting with it.

I did an audit and fixed what I believe are all instances of this URL
pattern behavior.  In the process, I fixed a few tests that were
unintentionally relying on the behavior.

Fixes #13082.
2019-08-26 20:51:49 -07:00
Tim Abbott e0f8228d6e auth: Add a test for the legacy /accounts/login/google/ mobile flow.
This is needed to return 100% URL coverage, and should also help
ensure we don't accidentally break the fix from
a43b231f90.
2019-08-26 20:51:49 -07:00
Tim Abbott a43b231f90 urls: Add backwards-compatibility URL for mobile Google login.
In bf14a0af4, we refactored the Google authentication system to use
the same code as GitHub auth, but neglected to provide a
backwards-compatible URL available for use by older versions of the
mobile apps.

Fixes #13081.
2019-08-26 20:14:04 -07:00
Tim Abbott c037ba49a0 i18n: Tag push notification subtitles for translation. 2019-08-26 15:02:04 -07:00
Tim Abbott 0ed0bb6828 messages: Add email/push notifications for wildcard mentions.
Historically, Zulip's implementation of wildcard mentions never
triggered either email or push notifications, instead being limited to
desktop notifications and the "mentions" counter.

We fix this just by plumbing the "wildcard_mentioned" flag through our
system.

Implements much of
https://github.com/zulip/zulip/issues/6040#issuecomment-510157264.
We're also now ready to seriously work on #3750.
2019-08-26 14:39:53 -07:00
Anders Kaseorg 43c8c720ef node_cache: Run yarn with --ignore-scripts.
As predicted in https://www.kb.cert.org/vuls/id/319816/, a malicious
worm is beginning to spread across the npm ecosystem through package
postinstall scripts.  Only instead of direct self-replicating code,
the replication vector is the temptation to monetize postinstall
scripts by polluting the console logs with paid advertisements.  The
effect will be the same unless we all put a stop to this while we
still can.

Apply the recommended VU#319816 workaround, which is to disable
lifecycle scripts when installing npm packages.  The only fallout is:

* node-sass can’t run because it uses compiled native code; we replace
  it with Dart Sass.
* phantomjs-prebuilt doesn’t download the binary at install time; we
  tell it to download it in run-casper.
* ttf2woff2 transparently falls back from native code to an Emscripten
  build.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:36:03 -07:00
Anders Kaseorg 37ece9b8e2 webpack: Replace node-sass with Dart Sass.
It doesn't require scripts to install, allowing us to migrate yarn to
the more secure --ignore-scripts option.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:35:26 -07:00
Anders Kaseorg 29abbdaa07 requirements: Upgrade pip-tools to 4.1.0 and pip to 19.2.3.
Closes #13068.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:33:18 -07:00
Vishnu KS d434c0ee88 slack: Remove unnecessary comments.
Remove comments that tries to explain code that is already readable.
Also remove some todo comments that has been already taken care of.
2019-08-26 14:10:19 -07:00
Vishnu KS 99d34fd11d slack: Rename default_channels to slack_default_channels. 2019-08-26 14:10:19 -07:00
Vishnu KS b919514f7f slack: Rename customprofilefield_id to custom_profile_field_id. 2019-08-26 14:10:19 -07:00
Vishnu KS c31355f9c1 slack: Rename custom_field_id_count to custom_profile_field_value_id_count. 2019-08-26 14:10:19 -07:00
Vishnu KS 138c659c97 slack: Rename slack_custom_field_name_to_zulip_custom_field_id.
Rename custom_field_map to
slack_custom_field_name_to_zulip_custom_field_id.
2019-08-26 14:10:19 -07:00
Vishnu KS 9560736d86 slack: Rename slack_user_id_to_custom_profile_fields.
Renames slack_user_custom_field_map to
slack_user_id_to_custom_profile_fields for readability.
2019-08-26 14:10:19 -07:00
Vishnu KS 01a51c8f4e slack: Rename added_recipient to slack_recipient_name_to_zulip_recipient_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 9d51a1b527 slack: Rename added_users to slack_user_id_to_zulip_user_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 3650f19692 slack: Lookup dir_name key in dict instead of in dict_keys.
No reason to do the lookup in O(n) when we can do
it in average O(1) time complexity.
2019-08-26 14:10:19 -07:00