Commit Graph

33200 Commits

Author SHA1 Message Date
Tim Abbott 83a012eaeb lint: Improve over-aggressive pattern botserver->Botserver pattern.
This is still not a great pattern, but this is sufficient to avoid it
firing on URLs of the form zulipbotserver.example.com.
2019-08-28 16:18:16 -07:00
Anders Kaseorg 9e481e353a .yarnrc: Set ignore-scripts true.
Follow up to #13065, to keep manual yarn invocations consistent with
our automated ones.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 16:15:54 -07:00
Rohitt Vashishtha 6fe99171c3 docs/botserver: Add basic troubleshooting help.
It could be a non trivial task ensuring that the Zulip server can
access the Botserver. These steps examine some of those cases.
2019-08-28 16:08:34 -07:00
Rohitt Vashishtha 7f3ee1b6b1 docs: Fix typo: Heroko -> Heroku. 2019-08-28 15:59:03 -07:00
Rohitt Vashishtha 16227ed23a docs/botserver: Explain how the botserver works.
Understanding the basic model can help people troubleshoot any
errors easily.
2019-08-28 15:59:03 -07:00
Tim Abbott c06d8129a3 custom profile fields: Add placeholder for custom URL.
This makes the feature useable for someone who hasn't used it before.
2019-08-28 15:43:35 -07:00
Yashashvi Dave 9e343f1a68 custom fields: Update create-field UI for default external account fields. 2019-08-28 15:39:20 -07:00
Yashashvi Dave 313003900a custom fields: Move field-type inputs first in create-field form.
As other data of field, such as field name, hint etc. are
relative to field type, this commit moves the field type
input to the first order in create field form in org settings.
2019-08-28 15:39:13 -07:00
Yashashvi Dave e05429b6a9 custom fields: Add default external account custom fields. 2019-08-28 15:35:53 -07:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
David Rosa 4afdfa6055 docs: Place warning at the top of every page in /latest/production.
- Added warning block to dev docs using _templates/layout.html
- Removed copy-pasted warning from install.md and requirements.md
- Removed unreleased tag in docs/conf.py that's no longer used

Useful ref: https://www.sphinx-doc.org/en/master/templating.html

Fixes #13056.
2019-08-28 15:31:47 -07:00
Wyatt Hoodes e64b5a2b88 data export: Fix success banner not clearing.
There was a bug where the success banner stuck
around even after the export completed.  We now
nicely fade and remove the banner upon a successful
population of the export in the table.

Fixes: #13045
2019-08-28 15:23:39 -07:00
Wyatt Hoodes 2a020fa6cc data export: Fix sort feature regression.
02cfb47 removed a couple HTML tags that were
being used to sort the table.  We fix this,
but disable filtering exports by marking the
input type as `hidden`.  We use this approach as
it seems `list_render` doesn't like an
undefined `opts.filter.element`, which is
what happens if we simply remove the `filter`
key.
2019-08-28 15:23:39 -07:00
Vinit Singh d09a80260b lint: Replace local variables named 'msgid' with 'message_id'.
Follow up of commit 2a1305d. Replace all local variables named 'msgid'
with 'message_id' in all JS and HTML files, and adds a linter rule for
it as well.

Resolves #12952.
2019-08-28 15:19:30 -07:00
Greg Price c6fb8c048a docs: Add troubleshooting entry for TLS <1.2 servers.
This corresponds to a user report at zulip/zulip-mobile#3596 .
2019-08-27 16:25:37 -07:00
Mateusz Mandera 6bd34afa0d models: Migration of UserMessage.id to bigint, part 2.
This requires part 1 (which can take hours to run but generally
doesn't require downtime) to be completed first.

This portion of the migration will require the server to be completely
down for a brief period; for chat.zulip.org with 250M UserMessage
rows, it took about 60s to run; that time will vary depending on
hardware details like whether the server has an SSD, but fundamentally
shouldn't be long.

Our upgrade-zulip and upgrade-zulip-from-git tools can apply this
migration correctly; nothing special needs to be done.

Fixes #13040.
2019-08-26 21:25:19 -07:00
Mateusz Mandera b008515d63 models: Migration of UserMessage.id to bigint, part 1.
As part of adding support for more than 2B UserMessage rows in a Zulip
server, we need to change UserMessage.id (a field we don't access but
is needed by Django) from an int to a bigint.  This commit is a series
of migrations which create a `bigint_id` column and populates it correctly.

This migration will take a long time to run; on chat.zulip.org (a
server with a lot of history), it took about 4 hours to complete.

How to migrate with minimal downtime:

1. Run `upgrade-zulip-from-git` through this commit.  It will install
migration 0238 and then more or less hang while applying migration
0239.  Once migration 0238 is completed, however, your server should
be able to be started back up safely while migration 0239 is running.

2. Run `/home/zulip/deployments/next/scripts/restart-server` in a
separate terminal to get Zulip running again.

3. When the `upgrade-zulip-from-git` command finishes, it will
automatically re-restart the Zulip server, leaving you in a consistent
state and ready to do part 2 of the migration.

A useful `manage.py shell` query for checking the state after this
commit is consistent is this:

    assert UserMessage.objects.exclude(bigint_id=F("id")).count() == 0

Part of #13040.
2019-08-26 21:14:15 -07:00
Tim Abbott 89aeefed76 urls: Tighten many unintentionally broad URL patterns.
Previously, several of our URL patterns accidentally did not end with
`$`, and thus ended up controlling just the stated URL, but actually a
much broader set of URLs starting with it.

I did an audit and fixed what I believe are all instances of this URL
pattern behavior.  In the process, I fixed a few tests that were
unintentionally relying on the behavior.

Fixes #13082.
2019-08-26 20:51:49 -07:00
Tim Abbott e0f8228d6e auth: Add a test for the legacy /accounts/login/google/ mobile flow.
This is needed to return 100% URL coverage, and should also help
ensure we don't accidentally break the fix from
a43b231f90.
2019-08-26 20:51:49 -07:00
Tim Abbott a43b231f90 urls: Add backwards-compatibility URL for mobile Google login.
In bf14a0af4, we refactored the Google authentication system to use
the same code as GitHub auth, but neglected to provide a
backwards-compatible URL available for use by older versions of the
mobile apps.

Fixes #13081.
2019-08-26 20:14:04 -07:00
Tim Abbott c037ba49a0 i18n: Tag push notification subtitles for translation. 2019-08-26 15:02:04 -07:00
Tim Abbott 0ed0bb6828 messages: Add email/push notifications for wildcard mentions.
Historically, Zulip's implementation of wildcard mentions never
triggered either email or push notifications, instead being limited to
desktop notifications and the "mentions" counter.

We fix this just by plumbing the "wildcard_mentioned" flag through our
system.

Implements much of
https://github.com/zulip/zulip/issues/6040#issuecomment-510157264.
We're also now ready to seriously work on #3750.
2019-08-26 14:39:53 -07:00
Anders Kaseorg 43c8c720ef node_cache: Run yarn with --ignore-scripts.
As predicted in https://www.kb.cert.org/vuls/id/319816/, a malicious
worm is beginning to spread across the npm ecosystem through package
postinstall scripts.  Only instead of direct self-replicating code,
the replication vector is the temptation to monetize postinstall
scripts by polluting the console logs with paid advertisements.  The
effect will be the same unless we all put a stop to this while we
still can.

Apply the recommended VU#319816 workaround, which is to disable
lifecycle scripts when installing npm packages.  The only fallout is:

* node-sass can’t run because it uses compiled native code; we replace
  it with Dart Sass.
* phantomjs-prebuilt doesn’t download the binary at install time; we
  tell it to download it in run-casper.
* ttf2woff2 transparently falls back from native code to an Emscripten
  build.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:36:03 -07:00
Anders Kaseorg 37ece9b8e2 webpack: Replace node-sass with Dart Sass.
It doesn't require scripts to install, allowing us to migrate yarn to
the more secure --ignore-scripts option.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:35:26 -07:00
Anders Kaseorg 29abbdaa07 requirements: Upgrade pip-tools to 4.1.0 and pip to 19.2.3.
Closes #13068.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:33:18 -07:00
Vishnu KS d434c0ee88 slack: Remove unnecessary comments.
Remove comments that tries to explain code that is already readable.
Also remove some todo comments that has been already taken care of.
2019-08-26 14:10:19 -07:00
Vishnu KS 99d34fd11d slack: Rename default_channels to slack_default_channels. 2019-08-26 14:10:19 -07:00
Vishnu KS b919514f7f slack: Rename customprofilefield_id to custom_profile_field_id. 2019-08-26 14:10:19 -07:00
Vishnu KS c31355f9c1 slack: Rename custom_field_id_count to custom_profile_field_value_id_count. 2019-08-26 14:10:19 -07:00
Vishnu KS 138c659c97 slack: Rename slack_custom_field_name_to_zulip_custom_field_id.
Rename custom_field_map to
slack_custom_field_name_to_zulip_custom_field_id.
2019-08-26 14:10:19 -07:00
Vishnu KS 9560736d86 slack: Rename slack_user_id_to_custom_profile_fields.
Renames slack_user_custom_field_map to
slack_user_id_to_custom_profile_fields for readability.
2019-08-26 14:10:19 -07:00
Vishnu KS 01a51c8f4e slack: Rename added_recipient to slack_recipient_name_to_zulip_recipient_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 9d51a1b527 slack: Rename added_users to slack_user_id_to_zulip_user_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 3650f19692 slack: Lookup dir_name key in dict instead of in dict_keys.
No reason to do the lookup in O(n) when we can do
it in average O(1) time complexity.
2019-08-26 14:10:19 -07:00
Vishnu Ks 1e5c49ad82 slack: Support importing shared channels. 2019-08-26 14:10:19 -07:00
Vishnu Ks e09a29f4d3 slack: Refactor get_slack_api_data to accept multiple query params. 2019-08-26 14:10:19 -07:00
Anders Kaseorg 51b2af9e5c docs: Shorten stackoverflow link.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:09:30 -07:00
Tim Abbott 73258efb16 development: Accept ToS for create user buttons.
We added default ToS for the development environment a few months
back; as a side effect, we now need to accept ToS when going through
the development environment registration flow, including for our
one-click account creation buttons.
2019-08-26 13:55:55 -07:00
Tim Abbott b8bfdebea1 onboarding: Mark the most recent 20 messages as unread.
After a new user joins an active organization, it isn't obvious what
to do next; this change causes there to be recent unread messages in
the stream sidebar for the user to click on to get a feel for what's
happening in the organization and experiment with Zulip.

Fixes #6512.
2019-08-26 13:51:52 -07:00
Greg Price 3c07c054b6 docs: Add troubleshooting entry for TLS issue on Android 7.0.
Learned this thanks to a user report and some debugging:
  https://chat.zulip.org/#narrow/stream/48-mobile/near/782174
2019-08-26 13:03:00 -07:00
Anders Kaseorg f1b91e577e requirements: Include packages that pip-tools considers unsafe.
It’s unclear why pip-tools considers these packages unsafe, and
excluding them from being pinned has resulted in nondeterministic
output that makes our test suite unhappy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 22:32:31 -07:00
Rohitt Vashishtha 8b443a25b8 markdown: Show link href if title is empty.
Fixes #6221.
2019-08-25 21:36:42 -07:00
Rohitt Vashishtha abe2dab88c markdown: Upgrade to use InlineProcessor for links.
This commit wraps up the major work that we held back when upgrading
py-markdown 2.6.11 to 3.0.1. Since we were making our custom changes
to the link syntax, at the time we stuck to using the old method of
parsing links. This lays the groundwork for further changes to our
link and image link handling, and brings us on par with upstream.

Also, we now better document the ways in which our link handling is
different from upstream.
2019-08-25 21:36:42 -07:00
Kanishk Kakar e4f0d3d79b notifications: Add 'none' to unread count options. 2019-08-25 21:29:10 -07:00
Yashashvi Dave 330d065a69 zerver/tests/test_custom_profile_data: Group tests cases. 2019-08-25 20:59:13 -07:00
Yashashvi Dave ed5cb6c28b custom profile fields: Extract field validation function. 2019-08-25 20:59:08 -07:00
Yashashvi Dave 166d4ce630 custom fields: Change field alert notifications style in user settings. 2019-08-25 20:57:27 -07:00
Tim Abbott 5e19546c14 messages: Fix unread_msgs accounting of wildcard mentions.
Previously, the unread_msgs data structure accounting (used for both
the web and mobile apps to determine the "Unread mentions" count
displayed in the UI) did not include wildcard mentions at all.

We fix this by adding the logic required to include properly that
data, with tests.  As discussed in #6040, it makes sense to include
muted streams and topics for the purpose of this calculation.

Fixes part of #6040.
2019-08-25 20:32:12 -07:00
Tim Abbott 625eb53b08 actions: Fetch presence idle data for non-basic mentions.
Apparently, get_active_presence_idle_user_ids, which is carefully
optimized to only fetch data for users who might actually need
notification processing, was only considering PMs and direct mentions,
not wildcard mentions or alert words.

This caused some pretty weird failure modes when working on adding
support for broader mention notifications, because users who had one
of these types of notifications would be treated as never
presence-idle, which was just confusing.

This is part of adding support for notifications for wildcard mentions
and alert words; it's worth merging this as an early commit because
the consequence of not doing this are very difficult to debug.
2019-08-25 19:58:33 -07:00
Tim Abbott 4e65f1dd2b test_event_queue: Clean up testing setup.
Rather than continually resetting the contents of an existing event
queue, we allocate a new one for each subtest.

We also fix a rather confusing bundle of comments.
2019-08-25 19:37:08 -07:00