Commit Graph

11025 Commits

Author SHA1 Message Date
Tim Abbott 8e7ce7cc79 python: Sort migrations/management command imports with isort.
This is a preparatory commit for using isort for sorting all of our
imports, merging changes to files where we can easily review the
changes as something we're happy with.

These are also files with relatively little active development, which
means we don't expect much merge conflict risk from these changes.
2020-01-14 13:07:47 -08:00
Tlazypanda 30ee0c2a49 invitations: Improve experience around reactivating users.
Previously, if you tried to invite a user whose account had been
deactivated, we didn't provide a clear path forward for reactivating
the users, which was confusing.

We fix this by plumbing through to the frontend the information that
there is an existing user account with that email address in this
organization, but that it's deactivated.  For administrators, we
provide a link for how to reactivate the user.

Fixes #8144.
2020-01-13 18:30:51 -08:00
Tim Abbott 571ce2f5cb populate_db: Fix handling of memcached flushing.
Our recent fixes to using the system's configured memcached settings
broke populate_db, because its hacky clear_database helper is called
with a hacked-up settings module.

We fix this by first moving this out-of-place code from models.py into
populate_db, and then saving the settings required to access memcached
so that we can use them in clear_database.

We also fix a mypy erorr in flush-memcached that matches the same
issue fixed in clear_database.
2020-01-13 18:05:21 -08:00
Anders Kaseorg 1ce15fba9c clear_database: Respect MEMCACHED_LOCATION.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:38:15 -08:00
Tim Abbott 79f18138f5 realm: Add private_message_policy setting.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.

Fixes #6617.
2020-01-13 12:20:42 -08:00
Mateusz Mandera d5ac1afce8 email_mirror: Check address usability in get_missed_message_address. 2020-01-12 20:43:51 -08:00
Mateusz Mandera 89046ea1a9 email_mirror: Give extract_and_validate a more descriptive name. 2020-01-12 11:30:18 -08:00
Mateusz Mandera 90a69ab24f email_mirror: Reuse exception messages in mirror_email_message. 2020-01-12 11:30:18 -08:00
Tim Abbott d8df1255d3 find_team: Send find team emails from the support address.
This is for consistency with the email's body, which claims replying
directly will work.
2020-01-08 21:55:34 -08:00
Mateusz Mandera 9f2b0c769f stream_recipient: Eliminate unnecessary queries.
We should take adventage of the recipient field being denormalized into
the Stream model. We don't need to make queries to figure out a stream's
recipient id, so we take advantage of that to eliminate some of
those redundant queries and simplify StreamRecipientMap.
2020-01-08 14:34:43 -08:00
Mateusz Mandera 786c235023 stream_recipient: Optimize query in populate_for_recipient_ids.
There's no reason to join with the Stream table, as Recipient.type_id is
the stream id.
2020-01-08 14:34:43 -08:00
shubhamgupta2956 92817e147d webhooks: Remove TrelloWebhookException classes.
This removes zerver/webhooks/trello/view/exceptions.py, which
contained legacy Trello webhook exception related classes.  We replace
them with UnexpectedWebhookEventType, which results in our standard
exception handling for unknown event types running (avoiding too-high
priority error logging).

Fixes #13467.
2020-01-07 18:02:40 -08:00
Hashir Sarwar 0cabacb8ab export: Fix data export parallelization.
This improves the approach of creating multiple parallel processes by
using subprocess.Popen() instead of run_parallel() and
subprocess.call() while exporting an organization's message
history.  This prevents forking twice for individual subprocess.

While this has some performance benefit, the main reason to fix this
is that it fixes an issue with the data export web UI introduced in
run_parallel forks exited).

Fixes #12904.
2020-01-07 13:23:18 -08:00
Mateusz Mandera b87cf22b33 email_mirror: Move send_to_mm_address code to process_missed_message.
process_missed_message did nothing other than calling
send_to_missed_message_address with the same arguments, so there's no
reason to have these as separate functions.
2020-01-07 13:03:32 -08:00
Mateusz Mandera c011d2c6d3 email_mirror: Migrate missed message addresses from redis to database.
Addresses point 1 of #13533.

MissedMessageEmailAddress objects get tied to the specific that was
missed by the user. A useful benefit of that is that email message sent
to that address will handle topic changes - if the message that was
missed gets its topic changed, the email response will get posted under
the new topic, while in the old model it would get posted under the
old topic, which could potentially be confusing.

Migrating redis data to this new model is a bit tricky, so the migration
code has comments explaining some of the compromises made there, and
test_migrations.py tests handling of the various possible cases that
could arise.
2020-01-07 13:03:22 -08:00
Mateusz Mandera 9077bbfefd models: Add MissedMessageEmailAddress class.
Preparatory commit for making the email mirror use the database instead
of redis for missed message addresses.

This model will represent missed message email addresses, which
currently have their data stored in redis.
The redis data will be converted and migrated into these models and
the email mirror will start using them in the main commit.
2020-01-07 12:46:55 -08:00
Steve Howell 630aadb7e0 bot_owner_id: Explicitly set bot_owner_id to None.
For cross realm bots, explicitly set bot_owner_id
to None.  This makes it clear that the cross realm
bots have no owner, whereas before it could be
misdiagnosed as the server forgetting to set the
field.
2020-01-07 12:33:14 -08:00
Mateusz Mandera 510bc60663 test_helpers: Set Recipient class attrs in use_db_models.
Model classes fetched through apps.get_model don't get methods or class
attributes. It's not feasible to add them to all these objects in
use_db_models, but Recipient.PERSONAL etc. are worth setting, since
doing that increases the range of functions that can successfully be
imported and called in test_migrations.py.
2020-01-03 16:56:58 -08:00
Mateusz Mandera a993604fae test_email_notifs: Clean up mocking.
These tests had a lot of very repetetive, identical mocking, in some
tests without even doing anything with the mocks. It's cleaner to put
the mock in the one relevant, common place for all the tests that need
it, and remove it from tests who had no use for the mocking.
2020-01-03 16:56:58 -08:00
Mateusz Mandera d691c249db api: Return a JsonableError if API key of invalid format is given. 2020-01-03 16:56:42 -08:00
Mateusz Mandera 72401b229f utils: Add a function to check if string can be an API key. 2020-01-03 16:56:42 -08:00
Mateusz Mandera 4f2897fafc cache: Validate keys before passing them to memcached.
Fixes #13504.

This commit is purely an improvement in error handling.

We used to not do any validation on keys before passing them to
memcached, which meant for invalid keys, memcached's own key
validation would throw an exception.  Unfortunately, the resulting
error messages are super hard to read; the traceback structure doesn't
even show where the call into memcached happened.

In this commit we add validation to all the basic cache_* functions, and
appropriate handling in their callers.

We also add a lot of tests for the new behavior, which has the nice
effect of giving us decent coverage of all these core caching
functions which previously had been primarily tested manually.
2020-01-03 16:56:42 -08:00
Mateusz Mandera e81aa740bc ldap: Protect against troublesome deactivations in ldap sync.
If ldap sync is run while ldap is misconfigured, it can end up causing
troublesome deactivations due to not finding users in ldap -
deactivating all users, or deactivating all administrators of a realm,
which then will require manual intervention to reactivate at least one
admin in django shell.
This change prevents such potential troublesome situations which are
overwhelmingly likely to be unintentional. If intentional, --force
option can be used to remove the protection.
2020-01-03 16:46:07 -08:00
Steve Howell 405a529340 server: Sort user_ids in recent PM conversations.
This change should prevent test flakes, plus
it's more deterministic behavior for clients,
who will generally comma-join the ids into
a key for their internal data structures.

I was able to verify test coverage on this
by making the sort reversed, which would
cause test_huddle_send_message_events to
fail.
2020-01-02 11:59:58 -08:00
Anders Kaseorg 8f281c4fc9 apply_event: Replace list comprehension with list.remove.
This should be about 4 times faster, saving something like half a
millisecond on each stream of 10000 subscribers.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-31 10:06:09 -08:00
Mateusz Mandera bbafced254 api docs: Advertise "topic" argument instead of "subject" on /messages.
They have the same meaning but we're transitioning away from the
"subject" terminology, so we should advertise "topic" in docs.
2019-12-30 17:22:46 -08:00
Mateusz Mandera e90866876c queue: Take advantage of ABC for defining abstract worker base classes.
QueueProcessingWorker and LoopQueueProcessingWorker are abstract classes
meant to be subclassed by a class that will define its own consume()
or consume_batch() method. ABCs are suited for that and we can tag
consume/consume_batch with the @abstractmethod wrapper which will
prevent subclasses that don't define these methods properly to be
impossible to even instantiate (as opposed to only crashing once
consume() is called). It's also nicely detected by mypy, which will
throw errors such as this on invalid use:

error: Only concrete class can be given where "Type[TestWorker]" is
expected
error: Cannot instantiate abstract class 'TestWorker' with abstract
attribute 'consume'

Due to it being detected by mypy, we can remove the test
test_worker_noconsume which just tested the old version of this -
raising an exception when the unimplemented consume() gets called. Now
it can be handled already on the linter level.
2019-12-28 10:52:17 -08:00
Mateusz Mandera ec209a9bc9 test_queue_worker: Extract a repetitive mock. 2019-12-28 10:52:13 -08:00
Mateusz Mandera a54640fc68 queue: Share exception handling code between loop and normal workers.
LoopQueueProcessingWorker can handle exceptions inside consume_batch in
a similar manner to how QueueProcessingWorker handles exceptions inside
consume.
2019-12-28 10:47:36 -08:00
Mateusz Mandera e559447f83 ldap: Improve logging.
Our ldap integration is quite sensitive to misconfigurations, so more
logging is better than less to help debug those issues.
Despite the following docstring on ZulipLDAPException:

"Since this inherits from _LDAPUser.AuthenticationFailed, these will
be caught and logged at debug level inside django-auth-ldap's
authenticate()"

We weren't actually logging anything, because debug level messages were
ignored due to our general logging settings. It is however desirable to
log these errors, as they can prove useful in debugging configuration
problems. The django_auth_ldap logger can get fairly spammy on debug
level, so we delegate ldap logging to a separate file
/var/log/zulip/ldap.log to avoid spamming server.log too much.
2019-12-28 10:47:08 -08:00
Tim Abbott 02169c48cf ldap: Fix bad interaction between EMAIL_ADDRESS_VISIBILITY and LDAP sync.
A block of LDAP integration code related to data synchronization did
not correctly handle EMAIL_ADDRESS_VISIBILITY_ADMINS, as it was
accessing .email, not .delivery_email, both for logging and doing the
mapping between email addresses and LDAP users.

Fixes #13539.
2019-12-15 22:59:02 -08:00
Vishnu KS 6901087246 install: Use crudini for storing value of POSTGRES_MISSING_DICTIONARIES.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.

Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
2019-12-13 12:05:39 -08:00
Tim Abbott 851eb1a6ee generate_test_data: Remove some useless type annotations.
One of these caused a parser error trying to run pyre on Zulip; the
other is just useless as the type can be inferred.
2019-12-13 11:52:23 -08:00
Mateusz Mandera 1926649dae migrations: Avoid triggering backend initalization in migration 0209.
Fixes #13528.
The email_auth_enabled check caused all enabled backends to get
initialized, and thus if LDAP was enabled the check_ldap_config()
check would cause an error if LDAP was misconfigured
(for example missing the new settings).
2019-12-13 10:54:05 -08:00
Tim Abbott 7ccc8373e2 bugdown: Fix logic for extracting attachment path_id.
In 3892a8afd8, we restructured the
system for managing uploaded files to a much cleaner model where we
just do parsing inside bugdown.

That new model had potentially buggy handling of cases around both
relative URLs and URLS starting with `realm.host`.

We address this by further rewriting the handling of attachments to
avoid regular expressions entirely, instead relying on urllib for
parsing, and having bugdown output `path_id` values, so that there's
no need for any conversions between formats outside bugdowm.

The check_attachment_reference_change function for processing message
updates is significantly simplified in the process.

The new check on the hostname has the side effect of requiring us to
fix some previously weird/buggy test data.

Co-Author-By: Anders Kaseorg <anders@zulipchat.com>
Co-Author-By: Rohitt Vashishtha <aero31aero@gmail.com>
2019-12-12 20:30:26 -08:00
Anders Kaseorg 8e37862b69 CVE-2019-19775: Close open redirect in thumbnail view.
This closes an open redirect vulnerability, one case of which was
found by Graham Bleaney and Ibrahim Mohamed using Pysa.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-12 17:29:20 -08:00
Tim Abbott 7bde70bb52 migrations: Batch fix_has_link_attribute migration.
This avoids risk of OOM issues on servers with relatively limited RAM
and millions of messages of history; apparently, fetching all messages
ordered by ID could be quite memory-intensive even with an iterator
usage model.

Fortunately, we have other migrations that already follow this pattern
of iterating over messages, so it's easy to borrow existing code to
make this migration run reasonably.
2019-12-12 15:29:49 -08:00
Tim Abbott 4901dc3795 url_preview: Fix parsing of open graph tags.
Our open graph parser logic sloppily mixed data obtained by parsing
open graph properties with trusted data set by our oembed parser.

We fix this by consistenly using our explicit whitelist of generic
properties (image, title, and description) in both places where we
interact with open graph properties.  The fixes are redundant with
each other, but doing both helps in making the intent of the code
clearer.

This issue fixed here was originally reported as an XSS vulnerability
in the upcoming Inline URL Previews feature found by Graham Bleaney
and Ibrahim Mohamed using Pysa.  The recent Oembed changes close that
vulnerability, but this change is still worth doing to make the
implementation do what it looks like it does.
2019-12-12 15:24:38 -08:00
Anders Kaseorg faa3ea0b8e oembed: Remove unsound HTML filtering.
The frontend now takes care of confining the HTML.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-12 15:24:38 -08:00
Tim Abbott cd3535ad0c migrations: Improve has_link migration.
This fixes a few minor issues with the migration:

* Skips messages with empty rendered_content, fixing an exception that
  affected 4 messages on chat.zulip.org.
* Accesses messages in order.
* Provides some basic output on the progress made.

This should make life substantially better for any organizations that
run into trouble with this migration, either due to it taking a long
time to run or due to any new exceptions.
2019-12-12 14:05:08 -08:00
Tim Abbott 9f223bb7c2 url_preview: Simplify path to oembed code. 2019-12-12 13:34:49 -08:00
Tim Abbott e7cf1112c8 notifications: Enable online push notifications by default.
For new user onboarding, it's important for it to be easy to verify
that Zulip's mobile push notifications work without jumping through
hoops or potentially making mistakes.  For that reason, it makes sense
to toggle the notification defaults for new users to the more
aggressive mode (ignoring whether the user is currently actively
online); they can set the more subtle mode if they find that the
notifications are annoying.
2019-12-12 13:04:10 -08:00
Vishnu KS 6c97a36355 install: Support remote database services like RDS.
Documentation and variable names edited by tabbott.
2019-12-12 12:59:45 -08:00
Tim Abbott f3c224058f models: Use unlimited .select_related() for Stream and DefaultStream.
Previously, these accesses used e.g. .select_related("realm"), which
was the only foreign key on the Stream model.  Since the intent in
these code paths is to attach the related models for efficient access,
we should just do that for all related models, including Recipient.
2019-12-12 12:13:07 -08:00
Mateusz Mandera 9a42a83e15 streams: Remove get_stream_recipients function and its uses.
With the recipient field being denormalized into the UserProfile and
Streams models, all current uses of get_stream_recipients can be done
more efficiently, by simply checking the .recipient_id attribute on the
appropriate objects.
2019-12-12 12:05:42 -08:00
Mateusz Mandera 01288ede9e recipients: Remove bulk_get_recipients function and its uses.
With the recipient field being denormalized into the UserProfile and
Streams models, all current uses of bulk_get_recipients can be done more
efficient, by simply checking the .recipient_id attribute on the
appropriate objects.
2019-12-12 12:00:13 -08:00
Tim Abbott 63fd7bdf57 actions: Simplify logic of get_recipient_from_user_profiles.
This just uses the early return pattern and a local variable to
produce somewhat more readable code.
2019-12-12 11:59:27 -08:00
Mateusz Mandera 9995dab095 messages: Save a database query in check_message code path.
The flow in recipient_for_user_profiles previously worked by doing
validation on UserProfile objects (returning a list of IDs), and then
using that data to look up the appropriate Recipient objects.

For the case of sending a private message to another user, the new
UserProfile.recipient column lets us avoid the query to the Recipient
table if we move the step of reducing down to user IDs to only occur
in the Huddle code path.
2019-12-12 11:49:01 -08:00
Mateusz Mandera 690dc7313d actions: Restore a misplaced comment to its correct position. 2019-12-11 18:46:33 -08:00
Mateusz Mandera 4eb629e276 auth: Use config_error instead of JsonableError in remote_user_sso. 2019-12-11 16:40:20 -08:00
Mateusz Mandera e955bfde83 auth: Check that the backend is enabled at the start of remote_user_sso. 2019-12-11 16:35:18 -08:00
Tim Abbott 299896b6ce notifications: Ignore mobile presence when sending notifications.
Previously, if the user had interacted with the Zulip mobile app in
the last ~140 seconds, it's likely the mobile app had sent presence
data to the Zulip server, which in turns means that the Zulip server
might not send that user mobile push notifications (or email
notifications) about new messages for the next few minutes.

The email notifications behavior is potentially desirable, but the
push notifications behavior is definitely not -- a private message
reply to something you sent 2 minutes ago is definitely something you
want a push notification for.

This commit partially addresses that issue, by ignoring presence data
from the ZulipMobile client when determining whether the user is
currently engaging with a Zulip client (essentially, we're only
considering desktop activity as something that predicts the user is
likely to see a desktop notification or is otherwise "online").
2019-12-11 16:05:35 -08:00
Tim Abbott 958f39a551 message_edit: Call check_attachment_reference_change unconditionally.
This removes the last of the messy use of regular expressions outside
bugdown to make decisions on whether a message contains an attachment
or not.  Centralizing questions about links to be decided entirely
within bugdown (rather than doing ad-hoc secondary parsing elsewhere)
makes the system cleaner and more robust.
2019-12-11 11:10:46 -08:00
Tim Abbott 60d307ac3f message: Add migrations to correct has_* fields.
This follows up on changes to correctly set has_link, has_attachment,
and has_image when rendering messages.

Fixes #12912.
2019-12-11 11:03:49 -08:00
Rohitt Vashishtha 3fbb050216 messages: Remove dependence on regex for claiming attachments.
This commit wraps up the work to remove basic regex based parsing
of messages to handle attachment claiming/unclaiming. We now use
the more dependable Bugdown processor to find potential links and
only operate upon those links instead of parsing the full message
content again.
2019-12-11 11:03:49 -08:00
Rohitt Vashishtha fe24f4ee65 messages: Remove update_calculated_fields method.
This infrastructure is no longer needed following reworking of how
has_link and friends work.
2019-12-11 11:03:49 -08:00
Rohitt Vashishtha 3892a8afd8 messages: Set has_attachment correctly using Bugdown.
Previously, we would naively set has_attachment just by searching
the whole messages for strings like `/user_uploads/...`. We now
prevent running do_claim_attachments for messages that obviously
do not have an attachment in them that we previously ran.

For example: attachments in codeblocks or
             attachments that otherwise do not match our link syntax.

The new implementation runs that check on only the urls that
bugdown determines should be rendered. We also refactor some
Attachment tests in test_messages to test this change.

The new method is:

1. Create a list of potential_attachment_urls in Bugdown while rendering.
2. Loop over this list in do_claim_attachments for the actual claiming.
   For saving:
3. If we claimed an attachment, set message.has_attachment to True.
   For updating:
3. If claimed_attachment != message.has_attachment: update has_attachment.

We do not modify the logic for 'unclaiming' attachments when editing.
2019-12-11 11:03:44 -08:00
Rohitt Vashishtha 4674cc5098 bugdown: Set message.has_image while rendering message. 2019-12-11 17:01:41 +05:30
dustinheestand 157c98de99 bugdown: Correctly set has_link attribute on messages.
Now autolinks and message edits affect the has_link attribute on messages.
2019-12-11 17:01:41 +05:30
Ryan Rehman 2589065405 tests: Rename invitor to inviter in test_signup and test_queue_worker.
"Inviter" seems to be preferred for the person who invites an invitee.
2019-12-10 17:22:32 -08:00
Ryan Rehman 6110bf96ca tests: Rename prereg_users to prereg_user in test_events.
This is a typo fix.
2019-12-10 17:21:04 -08:00
Rohitt Vashishtha 182503e5c0 bugdown: Move helper methods to InlineInterestingLinksProcessor.
add_a, add_oembed_data and add_embed are only called by
InlineInterestingLinksProcessor and this commit allows
these methods to access self.markdown object.
2019-12-10 15:35:00 -08:00
Mateusz Mandera 7ee54810a1 auth: Eliminate if/else block for PreregUser handling with/without SSO.
Both branches did very similar things, and the code is better having
common handling in all cases.
2019-12-10 20:16:21 +01:00
Mateusz Mandera 6dbd2b5fc3 auth: Merge RemoteUserBackend into external_authentication_methods.
We register ZulipRemoteUserBackend as an external_authentication_method
to make it show up in the corresponding field in the /server_settings
endpoint.

This also allows rendering its login button together with
Google/Github/etc. leading to us being able to get rid of some of the
code that was handling it as a special case - the js code for plumbing
the "next" value and the special {% if only_sso %} block in login.html.
An additional consequence of the login.html change is that now the
backend will have it button rendered even if it isn't the only backend
enabled on the server.
2019-12-10 20:16:21 +01:00
Mateusz Mandera a842968090 auth: Expand on the external_auth_method abstraction.
This commit builds a more complete concept of an "external
authentication method". Our social backends become a special case of an
external authentication method - but these changes don't change the
actual behavior of social backends, they allow having other backends
(that come from python-social-auth and don't use the social backend
pipeline) share useful code that so far only serviced social backends.
Most importantly, this allows having other backends show up in the
external_authentication_methods field of the /server_settings endpoint,
as well as rendering buttons through the same mechanism as we already
did for social backends.

This moves the creation of dictonaries describing the backend for the
API and button rendering code away into a method, that each backend in
this category is responsible for defining.

To register a backend as an external_authentication_method, it should
subclass ExternalAuthMethod and define its dict_representation
classmethod, and finally use the external_auth_method class decorator to
get added to the EXTERNAL_AUTH_METHODS list.
2019-12-10 20:16:21 +01:00
Rohitt Vashishtha 1229e69e9b bugdown: Reenable -,+ to begin a markdown list.
This commit has a side-effect that we also now allow mixed lists,
but they have different syntax from the commonmark implementation
and our marked output. For example, without the closing li tags:

  Input    Bugdown     Marked
-------------------------------------
         <ul>
- Hello    <li>Hello  <ul><li>Hello</ul>
+ World    <li>World  <ul><li>World
+ Again    <li>Again      <li>Again</ul>
* And      <li>And    <ul><li>And
* Again    <li>Again      <li>Again</ul>
         </ul>

The bugdown render is in line with what a user in #13447 requests.

Fixes #13477.
2019-12-09 16:13:02 -08:00
Nat1405 d5f005fd61 wildcard_mentions_notify: Add per-stream override of global setting.
Adds required API and front-end changes to modify and read the
wildcard_mentions_notify field in the Subscription model.

It includes front-end code to add the setting to the user's "manage
streams" page. This setting will be greyed out when a stream is muted.
The PR also includes back-end code to add the setting the initial state of
a subscription.

New automated tests were added for the API, events system and front-end.
In manual testing, we checked that modifying the setting in the front end
persisted the change in the Subscription model. We noticed the notifications
were not behaving exactly as expected in manual testing; see
https://github.com/zulip/zulip/issues/13073#issuecomment-560263081 .

Tweaked by tabbott to fix real-time synchronization issues.

Fixes: #13429.
2019-12-09 16:09:38 -08:00
Mateusz Mandera 792fbeea24 messages: Optimize check_message using recent denormalization. 2019-12-09 15:24:51 -08:00
Mateusz Mandera 1c5461663f users: Eliminate some unnecessary get_personal_recipient calls. 2019-12-09 15:24:35 -08:00
Mateusz Mandera 467833a974 streams: Eliminate some unnecessary get_stream_recipient calls. 2019-12-09 15:24:35 -08:00
Mateusz Mandera dda3ff41e1 messages: Optimize get_recent_private_conversations.
Previously, get_recent_private_messages could take 100ms-1s to run,
contributing a substantial portion of the total runtime of `/`.

We fix this by taking advantage of the recent denormalization of
personal_recipient into the UserProfile model, allowing us to avoid
the complex join with Recipient that was previously required.

The change that requires additional commentary is the change to the
main, big SQL query:
1. We eliminate UserMessage table from the query, because the condition
m.recipient_id=%(my_recipient_id)d
implies m is a personal message to the user being processed - so joining
with usermessage to check for user_profile_id and flags&2048 (which
checks the message is private) is redundant.
2. We only need to join the Message table with UserProfile
(on sender_id) and get the sender's personal_recipient_id from their
UserProfile row.

Fixes #13437.
2019-12-09 15:23:10 -08:00
Mateusz Mandera 8acfa17fe6 models: Add recipient foreign key in UserProfile and Stream.
This is adds foreign keys to the corresponding Recipient object in the
UserProfile on Stream tables, a denormalization intended to improve
performance as this is a common query.

In the migration for setting the field correctly for existing users,
we do a direct SQL query (because Django 1.11 doesn't provide any good
method for doing it properly in bulk using the ORM.).

A consequence of this change to the model is that a bit of code needs
to be added to the functions responsible for creating new users (to
set the field after the Recipient object gets created).  Fortunately,
there's only a few code paths for doing that.

Also an adjustment is needed in the import system - this introduces a
circular relation between Recipient and UserProfile. The field cannot be
set until the Recipient objects have been created, but UserProfiles need
to be created before their corresponding Recipients. We deal with this
by first importing UserProfiles same way as before, but we leave the
personal_recipient field uninitialized. After creating the Recipient
objects, we call a function to set the field for all the imported users
in bulk.

A similar change is made for managing Stream objects.
2019-12-09 15:14:41 -08:00
Tim Abbott 0c21855af8 models: Move Recipient model before UserProfile.
This is preparation for an upcoming partial denormalization that will
make UserProfile have a foreign key to Recipient.
2019-12-09 15:08:11 -08:00
Mateusz Mandera 586a5facc9 models: Add is_realm_admin and is_guest setters.
Fixes #13452.

The migration from UserProfile.is_realm_admin/UserProfile.is_guest in
e10361a832 broke our LDAP-based support
for setting a user's role via LDAP properties, which relied on setting
those fields.  Because the django-auth-ldap feature powering that only
supports booleans (and in any case, we don't want to expose constants
like `ROLE_REALM_ADMINISTRATOR` to the LDAP configuration interface),
it makes sense to provide setters for these legacy fields for
backwards-compatibility.

We lint against using these setters directly in Zulip's codebase
directly.  The issue with using these is that when changing user's
.role we want to create appropriate RealmAuditLog entries and send
events. This isn't possible when using these setters - the log entries
and events should be created if the role change in the UserProfile is
actually save()-ed to the database - and on the level of the setter
function, it's not known whether the change will indeed be saved.

It would have to be somehow figured out on the level of post_save
signal handlers, but it doesn't seem like a good design to have such
complexity there, for the sake of setters that generally shouldn't be
used anyway - because we prefer the do_change_is_* functions.

The purpose of this change is narrowly to handle use cases like the
setattr on these boolean properties.
2019-12-09 11:54:01 -08:00
Vishnu Ks 0296bba9ef openapi: Specify responses for users/me/subscriptions PATCH operation. 2019-12-06 11:19:08 -08:00
Vishnu KS c8ede33fc3 openapi: Specify securityScheme for the API in root level.
We used to specify the securityScheme for each REST operation seperately.
This is unecessary as the securityScheme can be specified in root level
and would be automatically applied to all operations. This also prevents
us accidentally not specifying the securityScheme for some operations and
was the case for /users/me/subscriptions PATCH endpoint. The root level
securityScheme can be also overriden in the operational level when
necessary.

swagger.io/docs/specification/authentication/#security
2019-12-06 11:19:08 -08:00
Vishnu KS e08d029dde docs: Use term operation instead of openapi in generate_curl_example.
The term operation makes more sense instead of openapi. OpenAPI
specs defines a unique operation as a combination of a path and a
HTTP method.
2019-12-06 11:19:08 -08:00
Tim Abbott 1465628c95 queue workers: Use self.queue_name in retry_event calls.
This just adds a bit of robustness if we ever end up renaming queues.
2019-12-04 10:08:48 -08:00
Mateusz Mandera 2b6cfbcf7b push_notifs: Handle more requests Exceptions in send_to_push_bouncer.
Closes #13294.
2019-12-04 09:58:22 -08:00
Mateusz Mandera 7d0444f903 push_notifs: Improve handling of errors when talking to the bouncer.
We use the plumbing introduced in a previous commit, to now raise
PushNotificationBouncerRetryLaterError in send_to_push_bouncer in case
of issues with talking to the bouncer server. That's a better way of
dealing with the errors than the previous approach of returning a
"failed" boolean, which generally wasn't checked in the code anyway and
did nothing.
The PushNotificationBouncerRetryLaterError exception will be nicely
handled by queue processors to retry sending again, and due to being a
JsonableError, it will also communicate the error to API users.
2019-12-04 09:58:22 -08:00
Mateusz Mandera 20b30e1503 push_notifs: Set up plumbing for retrying in case of bouncer error.
We add PushNotificationBouncerRetryLaterError as an exception to signal
an error occurred when trying to communicate with the bouncer and it
should be retried. We use JsonableError as the base class, because this
signal will need to work in two roles:
1. When the push notification was being issued by the queue worker
PushNotificationsWorker, it will signal to the worker to requeue the
event and try again later.
2. The exception will also possibly be raised (this will be added in the
next commit) on codepaths coming from a request to an API endpoint (for
example to add a token, to users/me/apns_device_token). In that case,
it'll be needed to provide a good error to the API user - and basing
this exception on JsonableError will allow that.
2019-12-04 09:58:22 -08:00
Mateusz Mandera 717e90dfeb test_push_notifications: Adjust mocking of requests.request.
requests.request is called in zerver/lib/remote_server.py, so these
mocks should be mocking it there, not in zerver.lib.push_notifications.
2019-12-04 09:58:22 -08:00
Mateusz Mandera 570de7fea4 test_queue_worker: Use MAX_REQUEST_RETRIES instead of a magic number.
The meaning is the same but it's clearer to use the involved constant
rather than putting in a magical number 4.
2019-12-04 09:58:21 -08:00
Mateusz Mandera ae8656e2c1 test_get_apns_client: Do cleanup in a finally: block. 2019-12-04 09:58:21 -08:00
Rohitt Vashishtha 85c669e366 markdown: Remove redundant checks from /me.
If a message begins with /me, we do not have any cases where the
rendered content would not begin with `<p>/me`. Thus, we can safely
remove the redundant checks both on the backend and frontend.
2019-12-03 17:17:10 -08:00
Rohitt Vashishtha 3df18c365d markdown: Bring /me python code up to date with frontend.
It appears we forgot to make identical changes to the backend
in #11089 while adding support for multiline /me messages,
resulting in any messages that didn't end in a paragraph getting
rendered as a regular message instead.

Fixes #13454.
2019-12-03 19:46:38 +05:30
Anders Kaseorg 7d5450917a test_messages: Fix list ordering flake in test_wildcard_mention.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-02 18:33:40 -08:00
Mateusz Mandera 74dd21c8fa register: Allow creating non-ldap users via social backends.
In configurations that use the ldap authentication backend and a social
backend, make it possible to create non-ldap users via the social backend.
2019-12-02 17:44:11 -08:00
Mateusz Mandera fcc91ae370 ldap: Disallow creating non-ldap accounts matching LDAP_APPEND_DOMAIN.
In configurations with LDAP_APPEND_DOMAIN, we don't want people creating
non-ldap accounts with emails matching the ldap domain.
So in the registration flow, if the email isn't found in LDAP, but
matches LDAP_APPEND_DOMAIN, we stop, rather than proceeding with account
creation. In case of emails not matching LDAP_APPEND_DOMAIN, we will
still continue to make a normal, non-ldap account.
2019-12-02 17:44:11 -08:00
Mateusz Mandera 82674b9b83 register: Improve handling of non-ldap users in LDAPPopulator configs.
The problem was that, for example, given a configuration of social
backend + LDAPPopulator, if a user that's not in ldap was being
registered, the Full Name field in the registration form would be
empty instead of getting prefilled with the name provided by the
social backend.

This fixes it - first we try to get the name from ldap. If that
succeeds, a form is created pre-filled with that name.  Otherwise, we
proceed to attempt to pre-fill with other means.

This also has a nice side effect of reorganizing most of the logic to
be more parallel between LDAP and other sources of name data.
2019-12-02 17:36:53 -08:00
Rohitt Vashishtha 68e93d2435 update-message: Use MentionData in the update_message_backend code.
This is a performance optimization, since we can avoid doing work
related to wildcard mentions in the common case that the message can't
have any.  We also add a unit test for adding wildcard mentions in a
message edit.
2019-12-02 12:12:35 -08:00
Rohitt Vashishtha bb42539b3f do_send_messages: Populate possible_wildcard_mentions from MentionData.
Fixes #13430.
2019-12-02 12:12:35 -08:00
Rohitt Vashishtha 9174c636ce bugdown: Store if message has wildcards in MentionData.
We also switch the underlying exctact_mention_text method to use
a regular for loop, as well as make the related methods return
tuples of (names, is_wildcard). This abstraction is hidden from the
MentionData callers behind mention_data.message_has_wildcards().

Concerns #13430.
2019-12-02 12:12:35 -08:00
David Rosa 1be4e10a2d docs: Explain link sharing in /api/upload-file.
Rewrittten by tabbott to clearly explain the security model, and add a
code example.
2019-12-02 12:01:46 -08:00
Tim Abbott 263ac0eb45 pm_conversations: Initialize using server data.
This simple change switches us to take advantage of the
server-maintained data for the pm_conversations system we implemented
originally for mobile use.

This should make it a lot more convenient to find historical private
message conversations, since one can effectively scroll infinitely
into the history.

We'll need to do some profiling of the backend after this is deployed
in production; it's possible we'll need to add some database indexes,
denormalization, or other optimizations to avoid making loading the
Zulip app significantly slower.

Fixes #12502.
2019-11-21 17:01:41 -08:00
Mateusz Mandera 67b6179df2 ldap: Fix error while updating a user registered in multiple realms.
Previously, the LDAP code for syncing user data was not
multiple-realm-aware, resulting in errors trying to sync data for an
LDAP user present in multiple realms.

Tweaked by tabbott to add some extended comments.

Fixes #11520.
2019-11-21 11:13:31 -08:00
Mateusz Mandera 06c2161f7e auth: Use zxcvbn to ensure password strength on server side.
For a long time, we've been only doing the zxcvbn password strength
checks on the browser, which is helpful, but means users could through
hackery (or a bug in the frontend validation code) manage to set a
too-weak password.  We fix this by running our password strength
validation on the backend as well, using python-zxcvbn.

In theory, a bug in python-zxcvbn could result in it producing a
different opinion than the frontend version; if so, it'd be a pretty
bad bug in the library, and hopefully we'd hear about it from users,
report upstream, and get it fixed that way. Alternatively, we can
switch to shelling out to node like we do for KaTeX.

Fixes #6880.
2019-11-21 10:23:37 -08:00
Mateusz Mandera 0c2cc41d2e CVE-2019-18933: Fix insecure account creation via social authentication.
A bug in Zulip's new user signup process meant that users who
registered their account using social authentication (e.g. GitHub or
Google SSO) in an organization that also allows password
authentication could have their personal API key stolen by an
unprivileged attacker, allowing nearly full access to the user's
account.

Zulip versions between 1.7.0 and 2.0.6 were affected.

This commit fixes the original bug and also contains a database
migration to fix any users with corrupt `password` fields in the
database as a result of the bug.

Out of an abundance of caution (and to protect the users of any
installations that delay applying this commit), the migration also
resets the API keys of any users where Zulip's logs cannot prove the
user's API key was not previously stolen via this bug.  Resetting
those API keys will be inconvenient for users:

* Users of the Zulip mobile and terminal apps whose API keys are reset
  will be logged out and need to login again.
* Users using their personal API keys for any other reason will need
  to re-fetch their personal API key.

We discovered this bug internally and don't believe it was disclosed
prior to our publishing it through this commit.  Because the algorithm
for determining which users might have been affected is very
conservative, many users who were never at risk will have their API
keys reset by this migration.

To avoid this on self-hosted installations that have always used
e.g. LDAP authentication, we skip resetting API keys on installations
that don't have password authentication enabled.  System
administrators on installations that used to have email authentication
enabled, but no longer do, should temporarily enable EmailAuthBackend
before applying this migration.

The migration also records which users had their passwords or API keys
reset in the usual RealmAuditLog table.
2019-11-21 10:23:37 -08:00
Tim Abbott f0fd812cc5 tornado: Add transitional code for sender_delivery_email.
This issue was introduced in 54e357e154.
2019-11-20 17:31:11 -08:00
Anders Kaseorg a681ca6cf5 queue: Update error callback signatures for Pika 1.1.
The expected signatures for these callbacks seem to have changed
somewhere in https://github.com/pika/pika/pull/1002.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-20 17:23:48 -08:00
Tim Abbott 1fe4f795af settings: Add notification settings checkboxes for wildcard mentions.
This change makes it possible for users to control the notification
settings for wildcard mentions as a separate control from PMs and
direct @-mentions.
2019-11-20 16:58:46 -08:00
Mateusz Mandera c3e83a0e6b push_notifications: Update link to google's upstream API docs.
Due to the migration to Firebase, the old link now gives 404. We replace
that with a working link to the migrated legacy API.
2019-11-20 10:50:24 -08:00
Hashir Sarwar cc56147c44 actions: Removed redundant assignment of message['mention_data']. 2019-11-20 10:49:01 -08:00
Tim Abbott 6407d0b1f9 push_notifications: Clear PushDeviceToken on API key change.
This includes adding a new endpoint to the push notification bouncer
interface, and code to call it appropriately after resetting a user's
personal API key.

When we add support for a user having multiple API keys, we may need
to add an additional key here to support removing keys associated with
just one client.
2019-11-19 15:37:43 -08:00
Anders Kaseorg 6717daf4a6 test_events: Fix apparent typo.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-19 15:34:05 -08:00
Tim Abbott 0338e8a6f6 send_custom_email: Add support for specifying reply-to. 2019-11-18 17:34:01 -08:00
Tim Abbott e266fb4343 management: Add command to send a custom email to users.
This allows us to email sets of users on a server with a nicely
formatted email similar to our onboarding emails, built off of a
Markdown template.

The code was based on send_password_reset_email, but it doesn't
replace that use case, since one cannot include special values like
password reset tokens in these emails.
2019-11-18 15:35:54 -08:00
Matheus Melo 39bd565941 test_realm: Automate tests for invalid integer values in Realm.
Fixes #13362.
2019-11-18 15:21:11 -08:00
Matheus Melo a890652bfd realm: Use check_int_in validator for enum format views.
This lets us remove some ugly semi-duplicated code.

Modified by tabbott to include some additional endpoints in the
changes.
2019-11-18 15:21:00 -08:00
Matheus Melo 678c3a89d0 validators: Add check_int_in validator.
This is a useful helper for our enum format fields where we want to
only allow a fixed list of integer values.
2019-11-18 15:14:56 -08:00
Matheus Melo 31558cb8b9 decorator: Extract require_user_group_edit_permission.
We move the check that the user is a member or admin inot this
decorator.

This name better communicates that this may do other checks beyond
just verifying the policy.
2019-11-18 15:13:29 -08:00
Matheus Melo 21ed834101 decorator: Extract OrganizationAdministratorRequired common exception.
This eliminates significant code duplication of error messages for
situations where an organization administrator is required.
2019-11-18 15:10:56 -08:00
Vishnu Ks 08103544cc tests: Remove upload-file from curl test exclude_list. 2019-11-18 12:23:38 -08:00
Vishnu Ks 76c610c953 tests: Remove upload-custom-emoji from curl test exclude_list. 2019-11-18 12:23:38 -08:00
Vishnu Ks 4c5e9e8eb0 tests: Add support for patching requestBody in curl example test. 2019-11-18 12:23:38 -08:00
Vishnu Ks 742de73bde tests: Remove update-notification-settings from curl test exclude_list. 2019-11-18 12:23:38 -08:00
Vishnu Ks c4525cceee tests: Remove get-user-groups from curl test exclude_list. 2019-11-18 12:23:38 -08:00
Vishnu Ks 3468764415 tests: Remove remove-linkifiers from curl test exclude_list. 2019-11-18 12:23:38 -08:00
Vishnu Ks 258089bdc3 tests: Remove add-linkifiers from curl test exclude_list. 2019-11-18 12:23:38 -08:00
Vishnu Ks ec53b8c574 tests: Unescape the html generated in curl example test.
Curl examples generated in test_generated_curl_examples_for_success
used to be html escaped. This commit removes the escaping in the test
since curl examples are not html escaped when run from terminal.
2019-11-18 12:23:38 -08:00
Vishnu Ks a478e861a2 tests: Remove create-user from curl test exclude_list.
The API test client uses an admin client since
0f64fe530667fd3b96a434842b124075dea84300
2019-11-18 12:23:38 -08:00
Tim Abbott 2e6e187c16 integrations: Deduplicate gogs and gitea integrations. 2019-11-18 12:08:09 -08:00
Jan Koscisz b88192d5bb integrations: Add Gitea integration.
Gitea integration adopted from Gogs integration with minor
adjusting. More events are now handled.

Fixes #13346
2019-11-18 11:55:24 -08:00
Jan Koscisz ace1503cb6 integrations: Add missing event handling to Gogs.
Issues event where not handled by Gogs integrations.
This added them to supported events.
2019-11-18 11:54:44 -08:00
Tim Abbott 4e421ebe12 scripts: Move inline-email-css from tools to scripts.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
2019-11-15 17:39:42 -08:00
Tim Abbott f34a40581f actions: Fix a few references to user_profile.email.
The only functional change here is for what is sent to the signups
queue worker, which at present isn't used in production anywhere.
2019-11-15 17:31:41 -08:00
Tim Abbott 6618cec9db logging: Switch various logging code paths to use user IDs.
This fixes EMAIL_ADDRESS_VISIBILITY_ADMINS support as well as being
more reliable/stable over time.
2019-11-15 17:24:01 -08:00
Tim Abbott 5edc5ae1fb create_test_bot: Use delivery_email for login.
The `email` field won't actually work if
EMAIL_ADDRESS_VISIBILITY_ADMINS is enabled.
2019-11-15 17:20:23 -08:00
Tim Abbott 0e317a8773 registration: Use delivery_email in logging code path.
This is essentially an assertion failure code path, so it doesn't
really matter, but it seems best to use the value that's the cause of
the problem here.
2019-11-15 17:18:19 -08:00
Tim Abbott caf4c7af8f get_user_by_id: Use delivery_email when finding cross-realm bots.
When actually looking for bots, it doesn't matter, but this feels more
consistent with the other lookup code.
2019-11-15 17:17:05 -08:00
Tim Abbott b85c9b0810 tornado: Use delivery_email in logging.
Eventually, we'll want to replace emails with user IDs here entirely,
but until we make that happen, we should at least use the same email
address present in our other logging.

I think we won't miss updating these in a future migration thanks to
mypy types.
2019-11-15 17:16:05 -08:00
Tim Abbott 7935b3cbd7 archive: Fix calculation of gravatar URL.
This is mostly dead code right now, but in case we revive it, I found
this error in an audit.
2019-11-15 17:14:44 -08:00
Tim Abbott 993ed9c2b1 tornado: Remove stale user_profile_email field.
Since years ago, this field hasn't been used for anything other than
some logging that would be better off logging the user ID anyway.

It existed in the first place simply because we weren't passing the
user_profile_id to Tornado at all.
2019-11-15 17:07:52 -08:00
Tim Abbott df90533eef home: Switch invalid pointer logging to by user ID. 2019-11-15 17:07:52 -08:00
Tim Abbott 093e9394dd find_account: Use delivery_email when searching emails.
Previously, this code wouldn't work with
EMAIL_ADDRESS_VISIBILITY_ADMINS.
2019-11-15 17:07:52 -08:00
Tim Abbott 3bc7ba1767 digest: Switch from emails to user IDs for logging.
This is better practice.
2019-11-15 17:07:52 -08:00
Tim Abbott bb64b0fa4d queue processors: Switch SignupWorker to logging user IDs.
This is a better setup than logging emails, especially with
EMAIL_ADDRESS_VISIBILITY_ADMINS.
2019-11-15 17:07:24 -08:00
Tim Abbott b14b18b76f soft_deactivation: Remove 'email' from logging.
The value wasn't correct with EMAIL_ADDRESS_VISIBILITY_ADMINS, and in
any case we really just need the user ID.
2019-11-15 17:06:51 -08:00
Tim Abbott 8b45756dd2 onboarding: Fix welcome emails not using delivery_email.
For organizations with EMAIL_ADDRESS_VISIBILITY_ADMINS, we were using
the wrong email address in the notice telling the user how to login in
the future.
2019-11-15 16:49:28 -08:00
Tim Abbott e124837cdc management commands: Fix incorrect use of user_profile.email.
All of these management commands should be interacting with
.delivery_email; this results in buggy behavior with
EMAIL_ADDRESS_VISIBILITY_ADMINS.
2019-11-15 16:43:51 -08:00
Vishnu Ks 0632d26e5e tests: Remove update-stream from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks be49dd4f30 tests: Remove delete-user-group from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu KS ee2d20ff3c tests: Remove update-user-group from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks 4b212efba6 tests: Remove create-user-group from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks 473d178fbc tests: Remove delete-stream from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks 4e15dabcfa tests: Remove remove-subscriptions from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu KS 3efe35e35b tests: Remove delete-message from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks 00455df7f9 tests: Use admin client for curl examples test. 2019-11-15 15:53:31 -08:00
Tim Abbott a41a4fcc33 onboarding: Use delivery_email in "new login" notifications.
Since we implemented EMAIL_ADDRESS_VISIBILITY_ADMINS, the intent is
that `delivery_only` should be used for accessing a user's actual
email address; with `email` used only in the Zulip API where we
haven't migrated to interacting with other users by ID.

This fixes a place we neglected to migrate.
2019-11-14 12:19:47 -08:00
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 40f4ead738 mypy: Upgrade from 0.720 to 0.730.
Fixes #13269.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 6fe5e44b35 settings: Define logging paths with, like, normal human variables.
This makes these variables available for type-checking.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:35 -08:00
Anders Kaseorg cafac83676 request: Tighten type checking on REQ.
Then, find and fix a predictable number of previous misuses.

With a small change by tabbott to preserve backwards compatibility for
sending `yes` for the `forged` field.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:35:55 -08:00
Anders Kaseorg b0a7b33f9b push_notifications: Declare token of type str, not bytes.
Declaring a CharField of type bytes made no sense.

Signed-off-by: Anders Kaseorg <andersk@zulipchat.com>
2019-11-12 23:21:20 -08:00
Gloria Elston f8855ca179 api: Remove legacy emoji reactions endpoint.
The original/legacy emoji reactions endpoints made use of HTTP PUT and
didn't have an API that could correctly handle situations where the
emoji names change over time.  We stopped using the legacy endpoints
some time ago, so we can remove them now.

This requires straightforward updates to older tests that were still
written against the legacy API.

Fixes #12940.
2019-11-12 13:07:06 -08:00
Tim Abbott ddd1a0eb00 actions: Convert do_delete_messages to take a Realm.
The function only used the user's realm anyway, so this is a cleaner
API.

This should also make it more convenient to permanently delete
messages manually, since one doesn't have to fetch a random user in
the realm in order to delete a message using the management shell.

No functional change.
2019-11-12 12:20:31 -08:00
Tlazypanda 6945ced76f slack import: Map Slack guest users to Zulip guests.
Slack's Single-User Guest and Multi-User Guest users should be
imported as Zulip guests during data import.

Fixes #13255.
2019-11-12 12:12:59 -08:00
Anders Kaseorg cce85f6ec7 dependencies: Upgrade katex from 0.10.2 to 0.11.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-11 16:26:31 -08:00
Tim Abbott 44f9ce92e9 bots: Fix rendering of bot owner fields in admin settings.
This fixes two regressions in 1946692f9a.

The first bug was actually introduced much earlier, namely that we
were not sending a `bot_owner_id` field at all for bot users without
an owner.  The correct behavior would have been send `None` for the
owner field.

The second bug was simply that we needed to update the webapp to look
for the `bot_owner_id` field, rather than an old email-address format
`bot_owner` field.

Thanks to Vinit Singh for reporting this bug.
2019-11-08 15:09:44 -08:00
Mateusz Mandera ed40d37e44 ldap: Fix realm_creation=True registration flow.
When creating realm with the ldap backend, the registration flow didn't
properly handle some things - the user wouldn't be set as realm admin,
initial subscriptions and messages weren't created, and the redirect
wasn't happening properly in the case of subdomains.
2019-11-08 14:01:45 -08:00
Mateusz Mandera 9ca26e91e0 ldap registration: Refactor, renaming auth_result to user_profile. 2019-11-08 14:01:45 -08:00
Mateusz Mandera fc4ad44dae registration: Refactor, reordering some conditional blocks. 2019-11-08 14:01:45 -08:00
Mateusz Mandera 071038f111 test_signup: Check that user is admin in RealmCreationTest. 2019-11-08 14:01:45 -08:00
Mateusz Mandera 3daec7783a ldap: Fix development environment configuration.
The state of the FAKELDAP setup for the dev env has fallen behind the
backend changes and updates to fakeldap (which implemented
SCOPE_ONELEVEL searches), as well as having some other minor issues.
This commit restore it to a working state and now all three config modes
work properly.
2019-11-08 14:00:24 -08:00
Tim Abbott 9692df4024 test_classes: Add sending_client argument to send_*_message.
This makes it possible to simlulate messages sent by specific clients,
rather than just "test suite".  Relevant for sending messages where
`message.sent_by_human()` is True.
2019-11-06 17:52:53 -08:00
Tim Abbott 509da6c9a6 get_recipient_info: Simplify user_ids_muting_topic handling.
Rather than subtracting sets in multiple places, it's simpler/cleaner
to just check which users are in the set when processing them.

This refactoring be helpful when we extend the get_recipient_info
logic to handle wildcard mentions as well.
2019-11-06 16:25:34 -08:00
Tim Abbott 43a965ff19 test_event_queue: Remove a stray print statement. 2019-11-06 16:25:34 -08:00
Mateusz Mandera b05a0d0177 social_backends: If no icon is to be displayed, set display_icon to None. 2019-11-05 15:44:07 -08:00
Mateusz Mandera c228f7ebcb email_notifs: Take full advantage of django_to_ldap_username.
django_to_ldap_username is now able to find the correct ldap username in
every supported type of configuration, so we can remove these
conditionals and use django_to_ldap_username in a straight-forward
manner.
2019-11-05 15:32:00 -08:00
Mateusz Mandera 8edbbe7b3c ldap: Make email search config obligatory without LDAP_APPEND_DOMAIN.
Having to account everywhere for both cases of having and not
having email search configured makes things needlessly complicated.
It's better to make the setting obligatory in configurations other than
LDAP_APPEND_DOMAIN.
2019-11-05 15:25:58 -08:00
Mateusz Mandera bfe800b11a ldap tests: Move test_ldap into test_auth_backends.
These tests belong more in test_auth_backends rather than deserving
their own separate file.
2019-11-05 15:25:58 -08:00
Mateusz Mandera bb3ddb9576 ldap tests: Put django_to_ldap tests in DjangoToLDAPUsernameTests.
test_auth_backends had a few random django_to_ldap_username tests laying
around, they belong in DjangoToLDAPUsernameTests.
2019-11-05 15:25:58 -08:00
Tim Abbott 54e357e154 avatars: Fix gravatar URLs with EMAIL_ADDRESS_VISIBILITY_ADMINS.
Previously, we were using user_profile.email rather than
user_profile.delivery_email in all calculations involving Gravatar
URLs, which meant that all organizations with the new
EMAIL_ADDRESS_VISIBILITY_ADMINS setting enabled had useless gravatars
not based on the `user15@host.domain` type fake email addresses we
generate for the API to refer to users.

The fix is to convert these calculations to use the user's
delivery_email.  Some refactoring is required to ensure the data is
passed through to the parts of the codebase that do the check;
fortunately, our automated tests of schemas are effective in verifying
that the new `sender_delivery_email` field isn't visible to the API.

Fixes #13369.
2019-11-05 12:33:20 -08:00
Tim Abbott fb59f98304 test_events: Fix handling of client_gravatar flag.
Previously, we weren't properly passing through the value of the
client_gravatar flag from the caller, resulting in buggy results if
the caller passed client_gravatar=False to do_test().

We happened to not have any uses of this before, but we're about to
add one.
2019-11-05 12:33:20 -08:00
Tim Abbott 399acea755 outgoing webhooks: Call finalize_payload on payloads.
Apparently, the refactor months ago that introduced finalize_payload
wasn't applied to the outgoing webhook code path, resulting in message
dicts with an unexpected format with no avatar_url and some extra
values that were intended to be internal details not relevant to
external clients.

Because this API is not widely used, we expect there to be little to
no impact of converting this back to matching the `get_messages`
interface, as it once was and has always been intended to be.

The one somewhat tricky detail is that we include both the `content`
and `rendered_content` fields, rather than asking the client to pick
which they want via the `apply_markdown` flag, because there is no
place for the client to configure that setting.
2019-11-05 12:26:33 -08:00
Tim Abbott 9bf947846e tests: Improve outgoing webhook testing strategy. 2019-11-05 11:49:00 -08:00
Tim Abbott aad99ce951 mattermost import: Fix handling of channels with no subscribers.
Previously, we skipped setting the list of subscribers to the channel,
which could result in problems if any messages had been posted there
in the past (e.g. because the channel used to have members, but now
doesn't).  It could be correct to skip importing dead channels
altogether, but probably simpler is to just set an empty subscriber list.
2019-11-04 18:10:37 -08:00
Tim Abbott dc682da47a mattermost: Handle replies to private messages.
Previously, our logic to handle Mattermost's "replies" feature didn't
copy the right fields for private messages, where `channel_members` is
included on the message body rather than a `channel` name.
2019-11-04 18:10:37 -08:00
Tim Abbott 47cf99a970 realm: Prevent setting invalid enum policy values.
As discussed in the comment, ideally these checks should be added
completely automatically, rather than needing to be manually added
every time we add a new setting.  But hopefully the example code for
all of the similar enums that this provides will at least provide some
help.
2019-11-03 17:11:56 -08:00
Tim Abbott 4066b58faa send_message: Remove long-obsolete comment about @require_login.
All of our view functions work this way, so it doesn't need a special
comment.
2019-11-03 16:52:28 -08:00
Matheus Melo c96762b7a9 settings: Add setting for who can edit user groups.
Fixes #12380.
2019-11-03 16:45:13 -08:00
Mateusz Mandera 849c925dc9 social_auth tests: Test registration with ldap populator enabled.
Finishes testing for the last remaining case and thus closes #13298.
2019-11-03 16:15:49 -08:00
Mateusz Mandera c1095474c6 social_auth tests: Extract some common logic in the long signup tests. 2019-11-03 16:15:49 -08:00
Mateusz Mandera 5aded51b73 register: Pre-populate Name in social backend flow.
By adding some additional plumbing (through PreregistrationUser) of the
full_name and an additional full_name_validated option, we
pre-populate the Full Name field in the registration form when coming
through a social backend (google/github/saml/etc.) and potentially skip
the registration form (if the user would have nothing to do there other
than clicking the Confirm button) and just create the account and log
the user in.
2019-11-03 16:15:48 -08:00
Mateusz Mandera 34a540bacb context: Rename social_backends to external_authentication_methods.
The main purpose of this is to make that name change happen in
/server_settings. external_authentication_methods is a much better, more
descriptive name than social_backends from API perspective.
2019-11-03 15:55:44 -08:00
Mateusz Mandera a62d084247 social_backends: Rename display_logo to display_icon. 2019-11-03 15:54:05 -08:00
Mateusz Mandera 5a39e70bce social_backends: Remove sort_order from social backend dicts.
These are returned through the API, at the /server_settings
endpoint. It's better to just return the list of dicts with a guarantee
of being sorted in the correct order, than to clutter things with the
sort_order field.
2019-11-03 15:51:49 -08:00
Mateusz Mandera 339f742578 test_signup: Add test for a non-covered case in accounts_register. 2019-11-03 15:51:19 -08:00
Anders Kaseorg f7245e9ec6 unminify: Update for webpack chunk splitting.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-01 18:25:30 -07:00
Mateusz Mandera 2e1529e879 registration: Fix logic disabling password requirement in the form.
This small block of code was over-indented. It should be run in this
part of the function unconditionally, not inside an "else" block.
We obviously want it to run regardless of whether
request.POST.get('from_confirmation')
is True or not.
2019-11-01 16:59:06 -07:00
Mateusz Mandera a4838d8b97 urls: Reorder where the accounts/login/google/ endpoint is defined.
Needed so that the google entry in social_backends in /server_settings
shows the new url rather than the legacy accounts/login/google/ url as
the login url.
2019-11-01 16:30:52 -07:00
Mateusz Mandera 29314f3195 api: Remove unused /get_auth_backends endpoint.
This legacy endpoint was designed for the original native Zulip mobile
apps, which were deprecated years ago in favor of the React Native
app.

It was replaced by /server_settings for active use years ago, so it's
safe to remove it now.
2019-11-01 12:30:59 -07:00
Vishnu Ks 2f5f23ecc6 tests: Remove get-presence from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks 7aabbe2f35 tests: Remove delete-queue from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks a99b96d9db tests: Remove get-events-from-queue from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks 4aae633d59 tests: Remove get-stream-topics from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks db294072be tests: Remove update-subscription-data from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks eb567abcec tests: Remove mark-as-read-bulk from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu KS 39f7d250bd tests: Remove update-message-flags from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu KS 31a50753f3 tests: Remove get-message-history from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu KS 93c003d2aa tests: Remove update-message from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu KS 8e9ccdf376 tests: Remove get-raw-message from curl test exclude_list. 2019-10-30 16:49:26 -07:00
Andrew Szeto b312001fd9 rabbitmq: Set a short TCP keepalive idle time on BlockingConnection.
The code comment explains this issue in some detail, but essentially
in Kubernetes and Docker Swarm systems, the container overlayer
network has a relatively short TCP idle lifetime (about 15 minutes),
which can lead to it killing the connection between Tornado and
RabbitMQ.

We fix this by setting a TCP keepalive on that connection shorter than
15 minutes.

Fixes #10776.
2019-10-30 16:15:44 -07:00
Rafid Aslam 447f74ae63 Upgrade pika to 1.1.*.
Upgrade pika to 1.1.* and make some changes accordingly
to comply with the new version.

Fixes #12899.
2019-10-29 17:01:12 -07:00
Mateusz Mandera f229839022 docs: Document social_backends in /server_settings.
This updates the docs for the /server_settings endpoint to document the
newly added social_backends structure.
2019-10-29 15:57:29 -07:00
Hemanth V. Alluri ac9008f564 openapi: Update the test and python code example for get-all-users.
This is following the change to the /users endpoint where we allow
an optional parameter "include_custom_profile_fields" which would
allow the client to request for users' custom profile fields along
with their other standard data.
2019-10-29 15:41:35 -07:00
Hemanth V. Alluri c1370547d5 events: Only send bot_type for bots and thus remove the for_api param. 2019-10-29 15:41:35 -07:00
Hemanth V. Alluri fa6bd42f4c docs: Update the example for /users for custom profile fields.
The previous example no longer gives a good enough idea of what the user
can expect when the `include_custom_profile_fields` boolean parameter is
set to true.
2019-10-29 15:41:35 -07:00
Hemanth V. Alluri 5b3e346369 users: Send custom profile fields with the /profile endpoint. 2019-10-29 15:41:35 -07:00
Mateusz Mandera b870816a75 saml: Sanity-check configuration in both login and signup codepaths. 2019-10-28 15:11:19 -07:00
Mateusz Mandera db29fcbbc4 auth: Add social_backends to /server_settings. 2019-10-28 15:11:19 -07:00
Mateusz Mandera 892d25faa1 auth: Change SAML login url scheme, enabling multiple IdP support.
The url scheme is now /accounts/login/social/saml/{idp_name} to initiate
login using the IdP configured under "idp_name" name.

display_name and display_logo (the name and icon to show on the "Log in
with" button) can be customized by adding the apprioprate settings in
the configured IdP dictionaries.
2019-10-28 15:09:42 -07:00
Mateusz Mandera 28dd1b34f2 auth: Refactor social login rendering.
login_context now gets the social_backends list through
get_social_backend_dicts and we  move display_logo customization
to backend class definition.

This prepares for easily adding multiple IdP support in SAML
authentication - there will be a social_backend dict for each configured
IdP, also allowing display_name and icon customization per IdP.
2019-10-28 15:06:26 -07:00
Jan Koscisz 12bd3c04c0 integration: Fix missing taiga test action.
Previously testing webhook from taiga site would result in error.
This adds missing 'test' action to taiga webhooks.

Fixes #13317.
2019-10-28 14:58:24 -07:00
Mateusz Mandera 8c065d1fcd ldap: Ensure django_to_ldap_username returns username that is in ldap.
This changes the way django_to_ldap_username works to make sure the ldap
username it returns actually has a corresponding ldap entry and raise an
exception if that's not possible. It seems to be a more sound approach
than just having it return its best guess - which was the case so far.
Now there is a guarantee that what it returns is the username of an
actual ldap user.

This allows communicating to the registration flow when the email being
registered doesn't belong to ldap, which then will proceed to register
it via the normal email backend flow - finally fixing the bug where you
couldn't register a non-ldap email even with the email backend enabled.

These changes to the behavior of django_to_ldap_username require small
refactorings in a couple of other functions that call it, as well as
adapting some tests to these changes. Finally, additional tests are
added for the above-mentioned registration flow behavior and some
related corner-cases.
2019-10-25 12:14:51 -07:00
Anders Kaseorg 453919bbc7 version: Only let `git describe` match tags beginning with a digit.
This will let us use other tags for things like `@zulip/shared`
versions.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-24 14:54:45 -07:00
Hemanth V. Alluri 1946692f9a users: Refactor get_members_backend endpoint to use get_raw_user_data.
Modify the get_raw_user_data method for use by the /users API endpoint
and then modify the /users endpoint to use it.
2019-10-23 14:50:26 -07:00
Hemanth V. Alluri dca990d60a events: Extract custom profile field logic from get_raw_user_data.
This is a simple refactor to move the code which gets custom profile
field values in the get_raw_user method to a separate function.
2019-10-23 14:36:07 -07:00
Hemanth V. Alluri 34849d7355 test_events: Fix a small typo for test_realm_bots_admin. 2019-10-23 14:35:32 -07:00
David Rosa bd3a8f1e96 docs: Merge front-end-build-process with html-css.
This merges the "Static asset pipeline" article as a new section
at the bottom of the "HTML and CSS" article.
2019-10-23 13:08:36 -07:00
Mateusz Mandera 869b57b7f7 ldap tests: Change TestQueryLDAP tests to use the test ldap directory.
Instead of mocking the _LDAPUser class, these tests can now take
advantage of the test directory that other ldap are using. After these
changes, test_query_email_attr also verifies that query_ldap can
successfully be used to query by user email, if email search is
configured.
2019-10-22 16:03:58 -07:00
Mateusz Mandera 1be2779515 tests: Add ldap_username() and ldap_password() method. 2019-10-22 16:03:58 -07:00
Mateusz Mandera 68f4cd1e94 ldap: Extract ldap user -> django username mapping logic to a function.
Fixes #11878

Instead of a confusing mix of django_auth_backed applying
ldap_to_django_username in its internals for one part of the
translation, and then custom logic for grabbing it from the email
attribute of the ldapuser in ZulipLDAPAuthBackend.get_or_build_user
for the second part of the translation,
we put all the logic in a single function user_email_from_ldapuser
which will be used by get_or_build of both ZulipLDAPUserPopulator and
ZulipLDAPAuthBackend.

This, building on the previous commits with the email search feature,
fixes the ldap sync bug from issue #11878.

If we can get upstream django-auth-ldap to merge
https://github.com/django-auth-ldap/django-auth-ldap/pull/154, we'll
be able to go back to using the version of ldap_to_django_username
that accepts a _LDAPUser object.
2019-10-22 16:02:23 -07:00
Mateusz Mandera 3699fe28f8 ldap: Use email search in django_to_ldap_username.
With this, django_to_ldap_username can take an email and find the ldap
username of the ldap user who has this email - if email search is
configured.

This allows successful authenticate() with ldap email and ldap password,
instead of ldap username. This is especially useful because when
a user wants to fetch their api key, the server attempts authenticate
with user_profile.email - and this used to fail if the user was an ldap
user (because the ldap username was required to authenticate
succesfully). See issue #9277.
2019-10-22 15:57:52 -07:00
Mateusz Mandera fea4d0b2be ldap: Do a proper search for email in email_belongs_to_ldap.
This fixes a collection of bugs surrounding LDAP configurations A and
C (i.e. LDAP_APPEND_DOMAIN=None) with EmailAuthBackend also enabled.

The core problem was that our desired security model in that setting
of requiring LDAP authentication for accounts managed by LDAP was not
implementable without a way to

Now admins can configure an LDAPSearch query that will find if there
are users in LDAP that have the email address and
email_belongs_to_ldap() will take advantage of that - no longer
returning True in response to all requests and thus blocking email
backend authentication.

In the documentation, we describe this as mandatory configuration for
users (and likely will make it so soon in the code) because the
failure modes for this not being configured are confusing.

But making that change is pending work to improve the relevant error
messages.

Fixes #11715.
2019-10-22 15:53:39 -07:00
Tim Abbott 87b43fdac0 tests: Fix whitespace around INVALID_STREAM_ID. 2019-10-21 22:14:46 -07:00
Vishnu KS e080b42fe5 tests: Move get_stream_id function to ZulipTestCase. 2019-10-21 16:56:48 -07:00
Vishnu KS 23036a9f40 confirmation: Set confirmation object realm attribute in realm reactivation.
The value of realm attribute in confirmation object used to be empty
before. We are not currently using the realm attribute of reactivation
links anywhere. The value of realm stored in content_object is currently
used.
2019-10-21 16:52:46 -07:00
Vishnu KS 139ebf387b support: Pass various realm functions as template context.
We currently have code to calculate the value of realm_icon_url,
admin_emails and default_discount in two diffrent places. With
the addition of showing confirmation links it would become three.
The easiest way to deduplicate the code and make the view cleaner
is by doing the calculations in template. Alternatively one can
write a function that takes users, realms and confirmations as
arguments and sets the value of realm_icon_url, admin_emails and
default_discount appropriately in realm object according to the
type of the confirmation. But that seems more messy than passing
the functions directly to template approach.
2019-10-21 16:52:46 -07:00
Tim Abbott b12d3d54c6 events: Fix documentation testing for /events.
Most of the failures were due to parameters that are not intended to
be used by third-party code, so the correct fix for those was the set
intentionally_undocumented=True.

Fixes #12969.
2019-10-21 16:50:10 -07:00
Tim Abbott c35472e025 test_openapi: Fix confused error messages for missing parameters.
These conditionals were backwards.
2019-10-21 16:26:08 -07:00
Tim Abbott c120ce6ecc events: Make it easy for test_openapi to find /events. 2019-10-21 16:26:00 -07:00
chgl bea9e41fbd webhooks: Add Harbor webhook integration. 2019-10-21 15:51:35 -07:00
Pragati Agrawal 857470c5ca events: Add delivery_email to user initial state data.
For admins, this adds `delivery_email` in addition to `email` for the
case when email address visibility is set to admins only.
2019-10-21 15:43:49 -07:00
Vishnu Ks 37c5f9b9c5 tests: Move add-linkifiers to correct position in exclude list. 2019-10-21 15:30:57 -07:00
Vishnu KS f2242c1487 docs: Make update-notification-settings use curl example system. 2019-10-21 17:10:35 +05:30
Mateusz Mandera bbf2474bd0 tests: setUp overrides should call super().setUp().
MigrationsTestCase is intentionally omitted from this, since migrations
tests are different in their nature and so whatever setUp()
ZulipTestCase may do in the future, MigrationsTestCase may not
necessarily want to replicate.
2019-10-19 17:27:01 -07:00
Vishnu KS 0af7aa8db3 docs: Make update-stream use curl example system.
new_name and description params should be valid JSON
strings. The format of these params are marked as
json so that the curl example genenrator can convert
them into json strings.
2019-10-18 13:45:30 -07:00
Vishnu Ks e96d96b6e4 docs: Add curl example to upload-file. 2019-10-18 13:45:30 -07:00
Vishnu Ks 9ac77a8734 docs: Make upload-custom-emoji use curl example system. 2019-10-18 13:45:30 -07:00
Mateusz Mandera 438c4b2935 tests: tearDown overrides should call super().tearDown().
ZulipTestCase subclasses that override tearDown() should call the parent
class tearDown() too, as it does some clean up of its own.
2019-10-18 13:36:59 -07:00
Mateusz Mandera 72dd834088 test_signup: Migrate ldap tests to the new format. 2019-10-17 17:10:39 -07:00
Mateusz Mandera 33050c733d test_email_notifications: Migrate ldap tests to the new format. 2019-10-17 16:49:53 -07:00
Mateusz Mandera e9df17884a test_settings: Migrate ldap tests to the new format. 2019-10-17 16:49:53 -07:00
Mateusz Mandera efba7290fd test_auth_backends: Migrate ldap tests to the new format. 2019-10-17 16:49:53 -07:00
Mateusz Mandera 8cbcf8cf45 tests: Add some logic to ZulipTestCase to prepare to migrate ldap tests. 2019-10-17 16:49:53 -07:00
Tim Abbott d364891894 ldap: Fix password prompt when configured only to populate data.
Previously, the logic for determining whether to provide an LDAP
password prompt on the registration page was incorrectly including it
if any LDAP authentication was backend enabled, even if LDAP was
configured with the populate-only backend that is not responsible for
authentication (just for filling in name and custom profile fields).

We fix this by correcting the conditional, and add a test.

There's still follow-up work to do here: We may still end up
presenting a registration form in situations where it's useless
because we got all the data from SAML + LDAP.  But that's for a future
issue.

This fixes a bug reported in #13275.
2019-10-17 14:46:46 -07:00
Rohitt Vashishtha 5fd0a121ea tests: Clean up and extend topic links tests in test_messages.
This is a follow-up to b69213808a.
We now actually send messages from the notification_bot, which
is the real usecase for this code.

Also, this cleans up the code and removes needless asserts like
`assertNotEqual(zulip_realm, lear_realm)` making the test easier
to read.
2019-10-16 17:39:09 -07:00
Vishnu Ks 1a0a282242 docs: Make delete-queue use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks f7add25cc2 docs: Make get-events-from-queue use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks 9d73578930 openapi: Fix narrow param example in /register POST endpoint. 2019-10-15 15:53:03 -07:00
Vishnu Ks 28e8df15a0 openapi: Fix fetch_event_types param example in /register POST endpoint. 2019-10-15 15:53:03 -07:00
Vishnu Ks 626c536b5e openapi: Fix event_types param example in /register POST endpoint. 2019-10-15 15:53:03 -07:00
Vishnu Ks 0b297fc092 docs: Make remove-linkifiers use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks 616502fd37 docs: Remove realm_logo and realm_night_logo from server-settings response.
They were removed in 16123c9a58
2019-10-15 15:53:03 -07:00
Vishnu Ks 6a5fdcabbd docs: Make delete-user-group use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks d1540f7e10 docs: Make update-user-group use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks c5d6be4df7 docs: Make create-user-group use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu KS 70e8f5ce8a docs: Make get-user-groups use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu KS 519b24061d docs: Update the response schema of get-presence.
The response no longer contains pushable and client keys.
2019-10-15 15:53:03 -07:00
Vishnu KS 503e65079a docs: Make get-presence use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks af83884f6f docs: Make create-user use curl example system. 2019-10-15 15:53:02 -07:00
Vishnu Ks ea06dc6ab3 docs: Make delete-stream use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 13bdfc0ef8 docs: Make get-stream-topics use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu KS bd6b41cb9d tests: Remove unused variable in test_curl_examples. 2019-10-15 15:40:44 -07:00
Vishnu Ks ad16c55c77 docs: Make remove-subscriptions use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 4364e31d69 docs: Make update-subscription-properties use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 89d82ab827 openapi: Improve description of subscriptions in /users/me/subscriptions. 2019-10-15 15:40:44 -07:00
Vishnu Ks c35ef2d390 bugdown: Add option to include only selected fields in generate_curl_example. 2019-10-15 15:40:44 -07:00
Vishnu Ks 3f9835ed07 docs: Make mark-as-read-bulk use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks f0b099dbc0 docs: Make update-message-flags use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 19ee518f2e docs: Make get-message-history use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks ed2e89f167 docs: Make delete-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 0dccebbe39 docs: Make update-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu KS 9bc3594f75 docs: Make get-raw-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks cb64fc8732 bugdown: Show example value for path params in curl example. 2019-10-15 15:40:44 -07:00
Vishnu Ks a03765bbe2 bugdown: Extract out function for getting openapi example value. 2019-10-15 15:40:44 -07:00
Rafid Aslam 718b70ec8b Rename `not_subscribed key` to `not_removed` in users/me/subscriptions.
Rename `not_subscibed_key` to `not_removed` in
`users/me/subscriptions` DELETE response.

Fixes #13277.
2019-10-13 10:30:34 +07:00
Mateusz Mandera a94b04dede test_mattermost_importer: Fix leftover references to message.pub_date.
The field is now called date_sent.
2019-10-11 04:07:12 +02:00
Tim Abbott d2970a56c2 lint: Remove some unused imports.
These were introduced in ae5bc92602.
2019-10-10 18:06:30 -07:00
Vishnu KS 1585ad7bf4 mattermost: Add support for exporting DMs and huddles. 2019-10-10 16:37:03 -07:00
Vishnu KS ae5bc92602 queue: Don't create confirmation objects twice during invite.
A confirmation object is already created when
do_send_confirmation_email is called just above.

Tweaked by tabbott to remove an unnecessary somewhat hacky database
query.
2019-10-10 16:19:42 -07:00
Mateusz Mandera 4dc3ed36c3 auth: Add initial SAML authentication support.
There are a few outstanding issues that we expect to resolve beforce
including this in a release, but this is good checkpoint to merge.

This PR is a collaboration with Tim Abbott.

Fixes #716.
2019-10-10 15:44:34 -07:00
Rohitt Vashishtha b69213808a messages: Render topic links in context of stream realm.
Priviously, we rendered the topic links using the msg.sender.realm.
This resulted in issues with Zulip's internal bots not having access
to the realm_filters of the destination stream's realm. For example,
sending a message via the email gateway or notification would not
linkify any realm filters that a user would expect them to.
2019-10-10 14:33:35 -07:00
Anders Kaseorg f25968f0ff exceptions: Fix AbstractEnum typing.
This will be needed for mypy 0.730.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-08 17:49:28 -07:00
Rishi Gupta 360cd7f147 remote data: Send RealmAuditLog data. 2019-10-08 17:27:29 -07:00
Rishi Gupta b86142089b billing: Sync realm activation events for on-prem billing. 2019-10-08 17:18:30 -07:00
Rishi Gupta 01bf99ec65 billing: Keep track of role count changes in RealmAuditLog. 2019-10-08 17:18:30 -07:00
Anders Kaseorg cc791afc54 requirements: Upgrade django-sendfile2.
django-sendfile2 now always sends a Content-Disposition header even if
it’s inline.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-08 17:15:00 -07:00
Vishnu KS cd06e0ab79 tests: Remove /messages/{message_id} from buggy endpoints. 2019-10-07 11:45:11 -07:00
Vishnu KS bfc9089124 tests: Remove /users/me/subscriptions/muted_topics from buggy endpoints.
Even though required attribute of stream and stream_id params is marked
false in openapi specification, the API expects atleast one of the
params to be set. There is no way to specify relationships like this
openapi and they dont seem to have any plan to implement this in future.

https://github.com/OAI/OpenAPI-Specification/issues/256
2019-10-07 11:42:52 -07:00
Vishnu KS e7419f815a tests: Remove /settings/notifications from buggy endpoints. 2019-10-07 11:42:52 -07:00
Vishnu KS 21ab8c542a bugdown: Cast enum elements to string in APIArgumentsTablePreprocessor.
So that enums other than of type string gets rendered without any error.
2019-10-07 11:42:52 -07:00
overide dd67b1b3bb topic: Change before bound limit from 2 to 7 days.
This limit was introduced in c588c79 as a part of the
feature and not due to performance crisis. So we are
increasing this limit to 7 days. Since topics tends to
naturally fizzle after day or two so 7 days limit
would be good enough.
2019-10-06 20:12:59 -07:00
Rishi Gupta 48dc1d1128 remote data: Refactor remote_server_post_analytics to be more generic.
One small change in behavior is that this creates an array with all the
row_objects at once, rather than creating them 1000 at a time.

That should be fine, given that the client batches these in units of
10000 anyway, and so we're just creating 10K rows of a relatively
small data structure in Python code here.
2019-10-06 16:55:41 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Shikhar Varshney fe5d975b33 settings: Add support for overriding APNS_TOPIC and ZULIP_IOS_APP_ID.
Documentation added by tabbott.
2019-10-05 22:53:59 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Mateusz Mandera f198475281 models: Migration of Message.pub_date to date_sent, part 1.
1. Apply migration 0243 to add date_sent column.
2. Apply migration 0244 to copy pub_date over to date_sent. Can be done
with the server running.
3. With the server down (for consistency between memory and
database state of Django objects), verify consistency with
Message.objects.exclude(date_sent=F("pub_date")).count() == 0
2019-10-05 19:01:34 -07:00
Vishnu Ks 30ad650afb bugdown: Don't show path params as data in curl example.
With test added by tabbott.
2019-10-05 18:17:36 -07:00
Vishnu Ks bfd5f4066a bugdown: Remove unecessary call to get_openapi_parameters. 2019-10-05 18:10:55 -07:00
Vishnu Ks dff62a240b bugdown: Rename openapi_example_params to openapi_params in generate_curl_example. 2019-10-05 18:10:55 -07:00
Vishnu Ks f4e3e6068f bugdown: Rename pocket to param in generate_curl_example. 2019-10-05 18:10:55 -07:00
Mateusz Mandera 94d5ca838f social auth tests: Extract prepare_login_url_and_headers method.
It will be reused in SAML tests in upcoming commits.
2019-10-01 16:39:12 -07:00
Mateusz Mandera 0e5f964363 social auth tests: Consistently refer to hamlet's name via self.name. 2019-10-01 16:38:50 -07:00
Tim Abbott c869a3bf82 upload: Fix browser caching of uploads with local uploads backend.
Apparently, our change in b8a1050fc4 to
stop caching responses on API endpoints accidentally ended up
affecting uploaded files as well.

Fix this by explicitly setting a Cache-Control header in our Sendfile
responses, as well as changing our outer API caching code to only set
the never cache headers if the view function didn't explicitly specify
them itself.

This is not directly related to #13088, as that is a similar issue
with the S3 backend.

Thanks to Gert Burger for the report.
2019-10-01 15:15:17 -07:00
Ivan Mitev 16c9d63056 emoji: Fix sort order of emoji choices.
The previous code for ensuring the sort order of emoji choices was
correct relied on an OrderedDict structure, which isn't guaranteed to
be preserved when passed to the frontend via JSON (in fact, it isn't,
since we converted the way page_params is passed to use
sort_keys=True).  Switch it to a list of dictionaries to correct this.

Fixes #13220.
2019-10-01 13:54:55 -07:00
Mateusz Mandera 4166c901ef do_update_user_custom_profile_data: Rename to ..._if_changed.
This adds clarity to the fact that the function no longer does
anything if the field values haven't changed.
2019-10-01 13:52:43 -07:00
Mateusz Mandera d66cbd2832 do_update_user_custom_profile_data: Don't notify if value not changed. 2019-10-01 13:50:14 -07:00
Tim Abbott 2b227d6b63 models: Add comments documenting our name field decisions. 2019-09-27 18:01:56 -07:00
Tim Abbott bf088519a7 api: Add avatar URL to the GET /profile endpoint.
This endpoint is legacy in a lot of ways, but it seems reasonable that
it should have these data.
2019-09-27 12:07:03 -07:00
Vishnu Ks 1c81538570 mattermost: Read fixures inside test cases instead of setUp. 2019-09-25 20:07:00 +05:30
Vishnu KS a21856c569 mattermost: Rename user_id to sender_user_id in process_raw_message_batch. 2019-09-25 20:06:47 +05:30
Vishnu KS 23d70bb685 mattermost: Rename get_recipient_id to get_recipient_id_from_receiver_name. 2019-09-25 20:06:04 +05:30
Vishnu KS a743991935 mattermost: Fix mirror dummy fixture data.
The channel where Snape sends message should be gryffindor-quidditch-team
and not slytherin-quidditch-team since he is mirror dummy in gryffindor.
2019-09-25 19:03:43 +05:30
Tim Abbott 7e0ea61b00 upload: Support S3-compatible S3 hosting providers.
Previously, we were hardcoding the domain s3.amazonaws.com.  Given
that we already have an interface for configuring the host in
/etc/zulip/boto.cfg (which in turn, automatically configures boto), we
just need to actually use the value configured in boto for what S3
hostname to use.

We don't have tests for this new use case, in part because they're
likely annoying to write with `moto` and there hasn't been a huge
amount of demand for it.  Since this doesn't regress existing S3
backend support, it seems worth merging.
2019-09-24 17:17:21 -07:00
Tim Abbott b8b0ae362c uploads: Only initialize S3 connection once in __init__.
This should be a mild performance optimization for the S3
authentication backend, since we aren't initializing unnecessary
duplicate connections.
2019-09-24 17:15:44 -07:00
Hemanth V. Alluri 98a33c0fda page_params: Fix list of config_options sent for incoming webhook bots.
This patches an issue in f37535044 where we mistakenly tried to send
the function as part of the page_params. Instead, we should just try
to send the list of configuration options (in their user displayable
form).
2019-09-24 16:04:51 -07:00
Laurel Michaels a67769091d docs: Add OpenAPI data for `/messages/{message_id}/reactions`.
This change adds the OpenAPI data needed to document the POST and
DELETE methods associated with this endpoint.

Descriptions edited slightly by tabbott.
2019-09-24 15:57:51 -07:00
Tim Abbott 96726c00ce export: Fix broken URLs in UI with S3 backend.
Apparently, the Zulip notifications (and resulting emails) were
correct, but the download links inside the Zulip UI were incorrectly
not including S3 prefix on the URL, making them not work.

While we're at this, we rewrite the somewhat convoluted previous
system for formatting the data export output.
2019-09-24 13:56:49 -07:00
Tim Abbott 8024b1179a bots: Fix bot email addresses with EMAIL_ADDRESS_VISIBILITY_ADMINS.
When using our EMAIL_ADDRESS_VISIBILITY_ADMINS feature, we were
apparently creating bot users with different email and delivery_email
properties, due to effectively an oversight in how the code was
written (the initial migration handled bots correctly, but not bots
created after the transition).

Following the refactor in the last commit, the fix for this is just
adding the missing conditional, a test, and a database migration to
fix any incorrectly created bots leaked previously.
2019-09-23 15:53:53 -07:00
Tim Abbott 6e5c99328a models: Extract a helper function for whether emails are realm-public.
This is also a useful preparatory refactor for having a user setting
controlling whether one's own email address is publicly available
within the organization.
2019-09-23 15:53:53 -07:00
Anders Kaseorg 9339402541 messages: Sort IDs in the by_streams query deterministically.
This fixes a test flake after upgrading SQLAlchemy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Anders Kaseorg cb0652071f zulip_otp_required: Avoid _user_is_authenticated compatibility shim.
This private shim is not needed with Django ≥ 1.10, and was removed
from django-otp 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Rishi Gupta 3646936cfb integrations: Advertise our native Zapier integration. 2019-09-22 18:47:42 -07:00
Tim Abbott 767a1b9f0c user_activity: Skip an unnecessary database query for new entries.
While new entries are an overwhelming minority of calls to this
function, it's still worth having a clean implementation.
2019-09-21 11:49:43 -07:00
Tim Abbott 1c73ce2450 user_activity: Use LoopQueueProcessingWorker strategy.
This should dramatically improve the queue processor's performance in
cases where there's a very high volume of requests on a given endpoint
by a given user, as described in the new docstring.

Until we test this more broadly in production, we won't know if this
is a full solution to the problem, but I think it's likely.  We've
never seen the UserActivityInterval worker end up backlogged without a
total queue processor outage, and it should have a similar workload.

Fixes #13180.
2019-09-21 11:48:24 -07:00
Tim Abbott 52a2b2c6a8 tests: Move user_activity test next to slow_query test.
This will conveniently put all the LoopQueueProcessingWorkers
together.
2019-09-21 11:47:14 -07:00
Tim Abbott f0d8951035 do_update_user_activity: Refactor to support passing a count.
We'll use this in upcoming commits.
2019-09-21 11:47:14 -07:00
Tim Abbott 5c960b3e0f user_activity: Make the queue processor a bit more efficient.
We don't actually need to go to the memcached (falling back to the
database) to fetch either user or client objects on every event.  For
user objects, we actually can just pass through the user ID
transparently; for client objects, we can use an in-process cache,
since the mapping of string to ID never changes.
2019-09-21 11:47:14 -07:00
Mateusz Mandera 3271235200 email_mirror: Ignore missed message email if the user isn't active. 2019-09-20 17:58:10 -07:00
Mateusz Mandera 21459ed193 test_email_mirror: Test missed message emails in a single ZulipTestCase.
With the way these tests are, it's unnecessary to have 3 separate
classes, and it makes it confusing to decide where to add a potential
additional mm email test.
2019-09-20 17:58:10 -07:00
Tim Abbott c19b7ec89d test_events: Remove unnecessary check_none validator.
This can be done cleanly using the existing equals(None).
2019-09-20 11:51:50 -07:00
Anders Kaseorg cf5a00d94b bulk_get_subscriber_user_ids: Sort each user list by ID.
This simple backwards-compatible change saves approximately 12% in the
compressed size of the chat.zulip.org page_params.  We can do much,
much better by changing the format, but this seems like a good
intermediate step.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:17:43 -07:00
Rishi Gupta 74d5d3db26 integration docs: Remove angle brackets from trello bash command.
If you accidentally leave them in, you'll get a confusing error message.
2019-09-19 17:48:16 -07:00
Yago González e1fbf6dddb actions: Make do_regenerate_api_key return the new key.
This way, the new API key can be fetched without needing to read it from
the UserProfile object.
2019-09-19 13:03:56 -07:00
Anders Kaseorg fbc2de157e templates: Move page_params to a <div> at the bottom of <body>.
In a gigantic realm where we send several MB of `page_params`, it’s
slightly better to have the rest of the `<body>` available to the
browser earlier, so it can show the “Loading…” spinner and start
fetching subresources.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:38:10 -07:00
Mateusz Mandera 935702b6ec auth: Rename any_oauth_backend_enabled to any_social_backend_enabled.
any_oauth_backend_enabled is all about whether we will have extra
buttons on the login/register pages for logging in with some non-native
backends (like Github, Google etc.). And this isn't about specifically
oauth backends, but generally "social" backends - that may not rely
specifically rely on Oauth. This will have more concrete relevance when
SAML authentication is added - which will be a "social" backend,
requiring an additional button, but not Oauth-based.
2019-09-19 12:35:27 -07:00
Anders Kaseorg fc0554fcb8 emoji: Finish removing leftover code from banned Emoji sets.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:08:52 -07:00
Vishnu Ks c4af0b7bc4 mattermost: Support importing messages without team name.
Mattermost doesn't place private messages within a particular team,
which is what this is needed for.
2019-09-18 11:57:37 -07:00
Vishnu Ks bf5f531e90 import_util: Support huddles in SubscriberHandler. 2019-09-18 11:53:13 -07:00
Rishi Gupta 2646d70cbd emails: Add realm name to footer of missed message emails.
Missed message emails for mentions come from streams and possibly orgs you
don't normally get missed message emails for, so they can be hard to place.
2019-09-17 17:45:59 -07:00
Anders Kaseorg fe7d814e8d team: Move contributors_list into page_params.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 7494f1600c templates: Move page_params from an inline script to the <body> dataset.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Wyatt Hoodes d2c3ddc765 test_events: Use stdout_suppressed in export test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes e37c91d195 test_realm_export: Use stdout_suppressed in tests.
From a testing perspective, calling `export_realm_wrapper`
prints unnecessary data to stdout.
2019-09-13 11:54:14 -07:00
Wyatt Hoodes e3595e4b3f test_slack_message_conversion: Use subTest for fixtures test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes dbaf6ac7e7 test_push_notifications: Remove fixtures print statement. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes 4a64a487d3 bugdown: Remove print statement from fixtures test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes 698b0f1b91 test_management_commands: Use subTest with help tests.
We remove the print statement, and instead use
subTest to improve debuggability.
2019-09-13 11:54:14 -07:00
Anders Kaseorg 78b018989e export: Refuse to overwrite an existing directory or tarball.
Previously, incorrectly passing an existing directory to the
`manage.py export --output` option would remove its contents without
warning.  Abort instead.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-12 16:04:55 -07:00
Anders Kaseorg 780ecb672b CVE-2019-16216: Fix MIME type validation.
* Whitelist a small number of image/ types to be served as
  non-attachments.
* Serve the file using the type that we validated rather than relying
  on an independent guess to match.

This issue can lead to a stored XSS security vulnerability for older
browsers that don't support Content-Security-Policy.

It primarily affects servers using Zulip's local file uploads backend
for servers running Ubuntu 16.04 Xenial or newer; the legacy local
file upload backend for (now EOL) Ubuntu 14.04 Trusty was not affected
and it has limited impact for the S3 upload backend (which uses an
unprivileged S3 bucket domain to serve files).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-11 15:46:36 -07:00
Thomas Ip 574c35c0b8 markdown: Render ordered lists using <ol> markup.
This brings us in line, and also allows us to style these more like
unordered lists, which is visually more appealing.

On the backend, we now use the default list blockprocessor + sane list
extension of python-markdown to get proper list markup; on the
frontend, we mostly return to upstream's code as they have followed
CommonMark on this issue.

Using <ol> here necessarily removes the behaviour of not renumbering
on lists written like 3, 4, 7; hopefully users will be OK with the
change.

Fixes #12822.
2019-09-08 16:42:20 -07:00
Wyatt Hoodes 4a176714e1 actions: Remove leftover print statement. 2019-09-08 15:36:30 -07:00
Tim Abbott 7e75f987df ldap: Fix logging of warning for deactivated users.
Also cleans up the interface between the management command and the
LDAP backends code to not guess/recompute under what circumstances
what should be logged.

Co-authored-by: mateuszmandera <mateusz.mandera@protonmail.com>
2019-09-08 09:35:23 -07:00
Tim Abbott d1a2784d52 ldap: Fix attempting to sync data for deactivated users.
The order of operations for our LDAP synchronization code wasn't
correct: We would run the code to sync avatars (etc.) even for
deactivated users.

Thanks to niels for the report.

Co-authored-by: mateuszmandera <mateusz.mandera@protonmail.com>
2019-09-08 09:35:23 -07:00
Mateusz Mandera 2ce2024bd7 ldap: Fix unintended user deactivation in case of connection failure.
Fixes #13130.

django_auth_ldap doesn't give any other way of detecting that LDAPError
happened other than catching the signal it emits - so we have to
register a receiver. In the receiver we just raise our own Exception
which will properly propagate without being silenced by
django_auth_ldap. This will stop execution before the user gets
deactivated.
2019-09-05 11:59:20 -07:00
Mateusz Mandera dfd6771237 stream settings: Use the .show-sender version of email address.
As requested in #13134.
2019-09-05 11:48:32 -07:00
Mateusz Mandera 307d8d8758 test_email_mirror: Better document the various address formats. 2019-09-05 11:48:32 -07:00
Vishnu KS 30440cf466 mattermost: Fix test flake in test_convert_channel_data.
So the reason 38f8cf612c seems
to be flaking is because the value of harry id switches between
1 and 2 in Xenial while in Bionic it would be fixed at 2. The
reason behind this is that Bionic ships with Python3.6 which
preserves dict insert order while Python3.5 that ships with Xenial
dont preserve the order. In initialize_stream_membership_dicts
we iterate user_data_map dict and the order in which the iteration
happens affects the ID of the users.
2019-09-05 11:20:33 -07:00
Puneeth Chaganti df134be235 webhooks/papertrail: Requests from Papertrail are not JSON requests.
Papertrail sends requests with the content type
`application/x-www-form-urlencoded`, with the payload parameter holding the
JSON body. This commit fixes the papertrail integration to use the payload
parameter in the request's POST data instead of trying to parse the
request's entire body as JSON.

Papertrail documentation here:
https://help.papertrailapp.com/kb/how-it-works/web-hooks#encoding
2019-09-04 13:16:17 -07:00
Wyatt Hoodes fc2574fdfd test_mattermost_importer: Add test coverage.
This is a rebase from a recent revert. A test
was failing non-deterministically relating to
`convert_channel_data` and use of an incorrect
`realm_id`.
2019-09-03 12:36:41 -07:00
Mateusz Mandera a7ec1a089c errors: Fix infinite recursion during memcached outages.
We have a very useful piece of code, _RateLimitFilter, which is
designed to avoid sending us a billion error emails in the event that
a Zulip production server is down in a way that throws the same
exception a lot. The code uses memcached to ensure we send each
traceback roughly once per Zulip server per 10 minutes (or if
memcached is unavailable, at most 1/process/10 minutes, since we use
memcached to coordinate between processes)

However, if memcached is down, there is a logging.error call internal
to the Django/memcached setup that happens inside the cache.set() call,
and those aren't caught by the `except Exception` block around it.

This ends up resulting in infinite recursion, eventually leading to
Fatal Python error: Cannot recover from stack overflow., since this
handler is configured to run for logging.error in addition to
logging.exception.

We fix this using a thread-local variable to detect whether we are
being called recursively.

This change should prevent some nasty failure modes we've had in the
past where memcached being down resulted in infinite recursion
(resulting in extra resources being consumed by our error
notifications code, and most importantly, the error notifications not
being sent).

Fixes #12595.
2019-09-03 11:44:39 -07:00
Tim Abbott 70c513a640 analytics: Fix logging for errors connecting to push bouncer.
There's no reason for this to be a category of error that emails the
server administrator, since there's a good chance that fixing it will
need to be done in the Zulip codebase, not administrator action.
2019-09-02 18:47:10 -07:00
Mateusz Mandera d70e1bcdb7 settings: Add FAKE_EMAIL_DOMAIN setting.
Fixes #9401.

This adds a FAKE_EMAIL_DOMAIN setting, which should be used if
EXTERNAL_HOST is not a valid domain, and something else is needed to
form bot and dummy user emails (if email visibility is turned off).
It defaults to EXTERNAL_HOST.

get_fake_email_domain() should be used to get this value. It validates
that it's correctly set - that it can be used to form valid emails.

If it's not set correctly, an exception is raised. This is the right
approach, because it's undesirable to have the server seemingly
peacefully operating with that setting misconfigured, as that could
mask some hidden sneaky bugs due to UserProfiles with invalid emails,
which would blow up the moment some code that does validate the emails
is called.
2019-08-30 14:59:00 -07:00
Tim Abbott 62c9ea7cf9 linkifiers: Fix problems with capture groups called "name".
Apparently, due to poor naming of the outer capture group we use to
separate the actual match from the surrounding whitespace (etc.) we
use to determine if the syntax is a possible linkifier start/end, if
you created a linkifier using "name" as the capture group, we'd try to
compile a pattern with two capture groups called "name", which would
500, preventing anyone from accessing the organization.
2019-08-30 09:36:14 -07:00
Anders Kaseorg c3e395b7d8 search: Reimplement ts_locs_array in pure PostgreSQL.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:59:12 -07:00
Anders Kaseorg d2d4800b1c test_narrow: Compile queries with the current SQLAlchemy dialect.
We’re about to start using PostgreSQL-specific syntax that can’t be
stringified without a specified dialect.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:58:01 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Yashashvi Dave e05429b6a9 custom fields: Add default external account custom fields. 2019-08-28 15:35:53 -07:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
Mateusz Mandera 6bd34afa0d models: Migration of UserMessage.id to bigint, part 2.
This requires part 1 (which can take hours to run but generally
doesn't require downtime) to be completed first.

This portion of the migration will require the server to be completely
down for a brief period; for chat.zulip.org with 250M UserMessage
rows, it took about 60s to run; that time will vary depending on
hardware details like whether the server has an SSD, but fundamentally
shouldn't be long.

Our upgrade-zulip and upgrade-zulip-from-git tools can apply this
migration correctly; nothing special needs to be done.

Fixes #13040.
2019-08-26 21:25:19 -07:00
Mateusz Mandera b008515d63 models: Migration of UserMessage.id to bigint, part 1.
As part of adding support for more than 2B UserMessage rows in a Zulip
server, we need to change UserMessage.id (a field we don't access but
is needed by Django) from an int to a bigint.  This commit is a series
of migrations which create a `bigint_id` column and populates it correctly.

This migration will take a long time to run; on chat.zulip.org (a
server with a lot of history), it took about 4 hours to complete.

How to migrate with minimal downtime:

1. Run `upgrade-zulip-from-git` through this commit.  It will install
migration 0238 and then more or less hang while applying migration
0239.  Once migration 0238 is completed, however, your server should
be able to be started back up safely while migration 0239 is running.

2. Run `/home/zulip/deployments/next/scripts/restart-server` in a
separate terminal to get Zulip running again.

3. When the `upgrade-zulip-from-git` command finishes, it will
automatically re-restart the Zulip server, leaving you in a consistent
state and ready to do part 2 of the migration.

A useful `manage.py shell` query for checking the state after this
commit is consistent is this:

    assert UserMessage.objects.exclude(bigint_id=F("id")).count() == 0

Part of #13040.
2019-08-26 21:14:15 -07:00
Tim Abbott 89aeefed76 urls: Tighten many unintentionally broad URL patterns.
Previously, several of our URL patterns accidentally did not end with
`$`, and thus ended up controlling just the stated URL, but actually a
much broader set of URLs starting with it.

I did an audit and fixed what I believe are all instances of this URL
pattern behavior.  In the process, I fixed a few tests that were
unintentionally relying on the behavior.

Fixes #13082.
2019-08-26 20:51:49 -07:00
Tim Abbott e0f8228d6e auth: Add a test for the legacy /accounts/login/google/ mobile flow.
This is needed to return 100% URL coverage, and should also help
ensure we don't accidentally break the fix from
a43b231f90.
2019-08-26 20:51:49 -07:00
Tim Abbott c037ba49a0 i18n: Tag push notification subtitles for translation. 2019-08-26 15:02:04 -07:00
Tim Abbott 0ed0bb6828 messages: Add email/push notifications for wildcard mentions.
Historically, Zulip's implementation of wildcard mentions never
triggered either email or push notifications, instead being limited to
desktop notifications and the "mentions" counter.

We fix this just by plumbing the "wildcard_mentioned" flag through our
system.

Implements much of
https://github.com/zulip/zulip/issues/6040#issuecomment-510157264.
We're also now ready to seriously work on #3750.
2019-08-26 14:39:53 -07:00
Vishnu KS d434c0ee88 slack: Remove unnecessary comments.
Remove comments that tries to explain code that is already readable.
Also remove some todo comments that has been already taken care of.
2019-08-26 14:10:19 -07:00
Vishnu KS 99d34fd11d slack: Rename default_channels to slack_default_channels. 2019-08-26 14:10:19 -07:00
Vishnu KS b919514f7f slack: Rename customprofilefield_id to custom_profile_field_id. 2019-08-26 14:10:19 -07:00
Vishnu KS c31355f9c1 slack: Rename custom_field_id_count to custom_profile_field_value_id_count. 2019-08-26 14:10:19 -07:00
Vishnu KS 138c659c97 slack: Rename slack_custom_field_name_to_zulip_custom_field_id.
Rename custom_field_map to
slack_custom_field_name_to_zulip_custom_field_id.
2019-08-26 14:10:19 -07:00
Vishnu KS 9560736d86 slack: Rename slack_user_id_to_custom_profile_fields.
Renames slack_user_custom_field_map to
slack_user_id_to_custom_profile_fields for readability.
2019-08-26 14:10:19 -07:00
Vishnu KS 01a51c8f4e slack: Rename added_recipient to slack_recipient_name_to_zulip_recipient_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 9d51a1b527 slack: Rename added_users to slack_user_id_to_zulip_user_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 3650f19692 slack: Lookup dir_name key in dict instead of in dict_keys.
No reason to do the lookup in O(n) when we can do
it in average O(1) time complexity.
2019-08-26 14:10:19 -07:00
Vishnu Ks 1e5c49ad82 slack: Support importing shared channels. 2019-08-26 14:10:19 -07:00
Vishnu Ks e09a29f4d3 slack: Refactor get_slack_api_data to accept multiple query params. 2019-08-26 14:10:19 -07:00
Tim Abbott 73258efb16 development: Accept ToS for create user buttons.
We added default ToS for the development environment a few months
back; as a side effect, we now need to accept ToS when going through
the development environment registration flow, including for our
one-click account creation buttons.
2019-08-26 13:55:55 -07:00
Tim Abbott b8bfdebea1 onboarding: Mark the most recent 20 messages as unread.
After a new user joins an active organization, it isn't obvious what
to do next; this change causes there to be recent unread messages in
the stream sidebar for the user to click on to get a feel for what's
happening in the organization and experiment with Zulip.

Fixes #6512.
2019-08-26 13:51:52 -07:00
Rohitt Vashishtha 8b443a25b8 markdown: Show link href if title is empty.
Fixes #6221.
2019-08-25 21:36:42 -07:00
Rohitt Vashishtha abe2dab88c markdown: Upgrade to use InlineProcessor for links.
This commit wraps up the major work that we held back when upgrading
py-markdown 2.6.11 to 3.0.1. Since we were making our custom changes
to the link syntax, at the time we stuck to using the old method of
parsing links. This lays the groundwork for further changes to our
link and image link handling, and brings us on par with upstream.

Also, we now better document the ways in which our link handling is
different from upstream.
2019-08-25 21:36:42 -07:00
Kanishk Kakar e4f0d3d79b notifications: Add 'none' to unread count options. 2019-08-25 21:29:10 -07:00
Yashashvi Dave 330d065a69 zerver/tests/test_custom_profile_data: Group tests cases. 2019-08-25 20:59:13 -07:00
Yashashvi Dave ed5cb6c28b custom profile fields: Extract field validation function. 2019-08-25 20:59:08 -07:00
Tim Abbott 5e19546c14 messages: Fix unread_msgs accounting of wildcard mentions.
Previously, the unread_msgs data structure accounting (used for both
the web and mobile apps to determine the "Unread mentions" count
displayed in the UI) did not include wildcard mentions at all.

We fix this by adding the logic required to include properly that
data, with tests.  As discussed in #6040, it makes sense to include
muted streams and topics for the purpose of this calculation.

Fixes part of #6040.
2019-08-25 20:32:12 -07:00
Tim Abbott 625eb53b08 actions: Fetch presence idle data for non-basic mentions.
Apparently, get_active_presence_idle_user_ids, which is carefully
optimized to only fetch data for users who might actually need
notification processing, was only considering PMs and direct mentions,
not wildcard mentions or alert words.

This caused some pretty weird failure modes when working on adding
support for broader mention notifications, because users who had one
of these types of notifications would be treated as never
presence-idle, which was just confusing.

This is part of adding support for notifications for wildcard mentions
and alert words; it's worth merging this as an early commit because
the consequence of not doing this are very difficult to debug.
2019-08-25 19:58:33 -07:00
Tim Abbott 4e65f1dd2b test_event_queue: Clean up testing setup.
Rather than continually resetting the contents of an existing event
queue, we allocate a new one for each subtest.

We also fix a rather confusing bundle of comments.
2019-08-25 19:37:08 -07:00
Mateusz Mandera 16479fa606 populate_db: Extract internal realm creation into a function. 2019-08-23 12:57:24 -07:00