Commit Graph

8416 Commits

Author SHA1 Message Date
Tim Abbott bdc95b5d72 slack webhooks: Eliminate unnecessary use of Service model.
The user ID of the bot user is a much more useful value to be sending
to the third-party API here.
2018-07-23 08:12:08 -07:00
Vishnu Ks 2b28042ddf models: Rename remaining audit log event types to past tense.
This makes the realm audit event type log entries more consistent.
2018-07-22 20:00:28 -07:00
Vishnu Ks 28167585b1 models: Rename several RealmAuditlog event types to past tense.
These event types haven't been used in production yet, and thus are
save to change without a migration.
2018-07-22 20:00:05 -07:00
amanagr 2d5ef27874 message-model: Add `zulipterminal` to sent_by_human clients.
Tweaked by tabbott to also more fully document this feature.
2018-07-22 12:32:55 -07:00
Anders Kaseorg a0293e8a86 zerver/lib/import_realm.py: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 195cc78470 zerver/lib/cache.py: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Steve Howell 5d0a113634 minor: Add assertion to assert_realm_values() helper. 2018-07-19 12:07:11 -04:00
Aditya Bansal 98a4e87e1d thumbor: Complete implementation of thumbnailing.
Various pieces of our thumbor-based thumbnailing system were already
merged; this adds the remaining pieces required for it to work:

* a THUMBOR_URL Django setting that controls whether thumbor is
  enabled on the Zulip server (and if so, where thumbor is hosted).

* Replaces the overly complicated prototype cryptography logic

* Adds a /thumbnail endpoint (supported both on web and mobile) for
  accessing thumbnails in messages, designed to support hosting both
  external URLs as well as uploaded files (and applying Zulip's
  security model for access to thumbnails of uploaded files).

* Modifies bugdown to, when THUMBOR_URL is set, render images with the
  `src` attribute pointing /thumbnail (to provide a small thumbnail
  for the image), along with adding a "data-original" attribute that
  can be used to access the "original/full" size version of the image.

There are a few things that don't work quite yet:
* The S3 backend support is incomplete and doesn't work yet.
* The error pages for unauthorized access are ugly.
* We might want to rename data-original and /thumbnail?size=original
  to use some other name, like "full", that better reflects the fact
  that we're potentially not serving the original image URL.
2018-07-15 00:39:41 +05:30
Sarah de947445ca event_queue: Add stream_email_notify.
This adds support to the event queue system for triggering
missed-message notifications (whether push or email) to support the
stream push notifications feature.
2018-07-14 12:19:33 +05:30
Sarah 73f5be65e8 zerver/lib/notifications: Add stream email notifications.
This modifies the logic for formatting outgoing missed-message emails
to support the upcoming stream email notifications feature (providing
a new format for the subject, etc.).
2018-07-14 12:19:33 +05:30
Tim Abbott 47df8c0c6e notifications: Elimininate now-unnecessary database query.
Because we're passing through the trigger for notifications to
do_send_missedmessage_events_reply_in_zulip, we don't need to go back
to the database to determine which messages actually mentioned the
user.
2018-07-14 12:19:33 +05:30
Tim Abbott dabf3667d1 notifications: Eliminate unnecessary at_mention context variable.
we already had a variable "mention" that did the exact same thing.
2018-07-14 12:19:33 +05:30
Tim Abbott bc861b4441 notifications: Pass trigger to do_send_missedmessage_events_reply_in_zulip.
This change converts our logic for determining whether the current
user was mentioned in a group of messages from the implicit "if it was
sent to a stream, it's a mention" to the explicit "we actually know
there was a mention in the message".  This is an important
prerequisite for our upcoming feature to support getting email
notifications for streams always (even without a mention).
2018-07-14 12:02:08 +05:30
Tim Abbott 58a7a390c8 event_queue: Call build_offline_notification unconditionally.
Previously, maybe_enqueue_notifications had this very subtle logic,
where it set the notice variable only inside the block for push
notifications, but then also used it inside the block for email
notifications.

This "worked", because previously the conditions for push
notifications were always true if the conditions for email
notifications were, but the code was unnecessarily confusing.  The
only good reason to write it this way is if build_offline_notification
was expensive; in fact, the most expensive thing it does is calling
time.time(), so that reason does not apply here.

This was further confusing, in that in the original logic, we relied
on the fact that push notification code path edited the "notice"
dictionary for further processing.

Instead, we just call it separately and setup the data separately in
each code path.
2018-07-14 12:01:59 +05:30
Tim Abbott a09ebf0551 event_queue: Remove confusing comment about rabbitmq.
Whatever RabbitMQ check this comment used to be next to, it isn't next
to anymore.
2018-07-14 12:00:37 +05:30
Tim Abbott 08ed1b4605 notifications: Set trigger on offline email notifications.
This data will be required for correctly implementing the upcoming
stream_push_notify feature; it also helps support cleaning up the code
for the existing stream mentions logic.
2018-07-14 11:57:28 +05:30
Tim Abbott b0c393cf97 notifications: Format missed_messages argument as a dict.
Because in upcoming commits, we'll want to pass additional per-message
data into do_send_missedmessage_events_reply_in_zulip, we need to
expand the format for how we represent messages to account for that.
2018-07-14 11:34:04 +05:30
Aditya Bansal 2e837b1407 archives: Change the topic/<topic_name> code path to use topics prefix.
We do this for maintaining consistency in naming of the endpoints.
2018-07-14 10:39:35 +05:30
Tim Abbott bf1ad714da actions.py: Refactor generate_topic_history_from_db_rows for clarity.
This refactors the generate_topic_history_from_db_rows function to not
depend upon the assumption of rows passed as parameter to be sorted in
reverse order of max_message_id field.

Additionally, we add sorting and some tests that verify correct
handling of these cases.
2018-07-14 10:11:08 +05:30
Aditya Bansal 63ec8b08b8 archives: Add endpoint to fetch topic history of web public streams.
In this commit we add a new endpoint so as to have a way of fetching
topic history for a given stream id without having to be logged in.
This can only happen if the said stream is web public otherwise we
just return an empty topics list. This endpoint is quite analogous
to get_topics_backend which is used by our main web app.

In this commit we also do a bit of duplication regarding the query
responsible for fetching all the topics from DB. Basically this
query is exactly the same as what we have in the
get_topic_history_for_stream function in actions.py. Basically
duplicating now is the right thing to do because this query is
really gonna change when we add another criteria for filtering
messages which is:
Only topics for messages which were sent during the period the
corresponding stream was web public should be returned.
Now when we will do this, the query will change and thus it won't
really be a code duplication!
2018-07-14 09:51:37 +05:30
Eeshan Garg 67045d65a9 webhooks/gitlab: Include title in Issue Hook event messages.
We already include the issue title in the topic. But if one chooses
to group all gitlab notifications under one topic, the message body
is misleading in the sense that only the Issue ID and the description
are displayed, not the title, which isn't super helpful if the topic
doesn't tell you the title either.

I think we should err on the side of always including the title in
the main message body, which is what this commit does.

Fixes #9913.
2018-07-14 09:38:11 +05:30
Harshit Bansal f636882e04 build_emoji: Migrate to use `emoji_names.py` file.
This migrates Zulip to use a dramatically better set of names and
aliases for our emoji set, defined in emoji_names.py (which is in turn
manually generated from our hand-curated CSV file).

This should significantly improve the experience of using Zulip's
emoji picker and emoji typeahead for finding what one is looking for.
2018-07-13 21:18:02 +05:30
Tim Abbott ab5fef87e5 models: Document RealmAuditLog. 2018-07-13 18:38:33 +05:30
Aastha Gupta dfde4fac85 invitations: Send 'invites_changed' event for invitations events.
Fixes #7665

In case of invitation events, 'invites_changed' event without
any real payload is sent to all the realm admins and the user.
The event is handled by reloading the list to view recent changes.

Commit tweaked by shubhamdhama:
* Send an `invite_changed` event when an user accept an invite.
    Also, added the test for the same.
* No need to delete the invite list in frontend, current logic
    handles the case when the invite data is changed properly.
* Extracted the common logic for sending an event into
    `notify_invites_changed`.
2018-07-13 18:00:06 +05:30
Tim Abbott 40efac2193 api docs: Clean of high levels of code nesting.
An early continue here makes the code a lot more readable.
2018-07-13 17:39:20 +05:30
Yago González 1cdd6862b7 api docs: Display example arguments as JSON.
Some of the arguments in our REST API have to be sent as JSON objects,
which only accept double quotes for strings.

If we display the examples as normal Python objects, the syntax would be
quite similar but it would use simple quotes, which is invalid JSON (and
isn't accepted by the server).

That's why all the examples should be JSON-serialized in order to comply
with the API's requirements.
2018-07-13 17:36:41 +05:30
Yago González c2f3c4282b api docs: Show message when there are no arguments.
Until now, we were displaying an empty "Arguments" section in the REST
API docs whenever an endpoint didn't use input arguments.

In the case of OpenAPI-based docs, that was also annoying because it
required removing the {generate_api_arguments_table|...} template tag or
leaving an empty "parameters" field in zulip.yaml.

After this, we show a paragraph indicating that the endpoint doesn't
need arguments under the "Arguments" section.
2018-07-13 17:36:40 +05:30
Yago González e6631db6b6 api docs: Raise exception on missing argument file.
If the argument table generator isn't able to reach a file that is
supposed to read, the two most likely causes are:

- The source .md documentation file that is requesting the table has a
typo in the path.
- The file with the arguments isn't there, for some reason.

In either case, we don't want the server to fail silently-ish and
display the docs as if there was no arguments for that endpoint. That's
why the most logic thing to do is to raise an exception and let the
admins know that there's something wrong.
2018-07-13 17:33:06 +05:30
Anders Kaseorg f75c0e29f4 HTML validation: Remove obsolete attributes from <script>, <style>.
<script charset=…>, <script type=…>, and <style type=…> are “obsolete
but conforming” in HTML5.  They make the validator.nu output noisier
and real problems a little harder to find.

(type was required in HTML 4, which is not relevant to us.)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-13 10:27:23 +05:30
Rhea Parekh fff0f64dea tests: Refactor tests for importing huddles. 2018-07-13 01:14:42 +05:30
Rhea Parekh ee830ddee9 tests: Fix mypy bug in test_import_realm. 2018-07-13 01:14:42 +05:30
Yago González 0e135b69f9 alert words: Return the updated list when adding/removing words.
POST and DELETE operations in /users/me/alert_words may leave the
user's list of alert words in an unknown state: POSTing adds words to a
list that the client may not know from the begining, and the same with
DELETE.

Replying with the current status of the alert words list is the best way
of letting the client alter the list and knowing its contents after
being updated with a single query.

This is especially useful taking into account that POSTing words that
were already present and DELETing non-existing words both produce a
successful response.

An extra test has been added to avoid leaving GET /users/me/alert_words
too untested.
2018-07-13 01:01:00 +05:30
Yago González 3b21b17a35 alert words: Make alert_words required for POST and DELETE.
Querying an endpoint with no information (thus a noop) and it producing
a successful response doesn't seem to be expected.

Given the case that the client makes such query with no content it will
probably be unintentional and the API should let them know about it.
2018-07-13 01:01:00 +05:30
Rhea Parekh e9884916c9 import: Support import of huddles.
For importing huddles we have to have unique huddle hashes.
Huddle hashes are extracted from the list of users participating
in a huddle. So to extract these user ids, we first use huddle
id to getting the matching recipient, and then we use subscription
to get the user ids from the recipient id.

Added tests for the same (tests slightly tweaked by tabbott).
2018-07-12 19:06:52 +05:30
Yago González fa1a695e08 test-api: Fix broken check for get_members.
The tests for GET /users were looking for a specific user, asuming that
it would always be in the same position. Since the users' sorting isn't
guaranteed in any way, this can lead to errors in the tests.

Now we make sure the user we grab from the list is the one we need by
checking its email address.

This is just a hotfix that addresses the short-term problem: we have
already made some efforts to make sure these tests are more
deterministic, and now we only need to finish the migration of the old
enpoints to the new system as a long-term solution.
2018-07-12 18:54:06 +05:30
Tim Abbott 5ae90d60d7 tests: Fix a failing test in test_users.py.
This small change should have been included in
d1cda29fb4.
2018-07-12 15:05:16 +05:30
Sarah d1cda29fb4 API: Add stream email notification setting plumbing.
This is all the plumbing that makes it possible to enable the
stream_email_notifications setting via the Zulip API.  The flag still
doesn't do anything yet, but this is a nice checkpoint along the way
to implementing this feature.
2018-07-12 13:39:14 +05:30
Tim Abbott aa5ca4e549 actions: Remove unnecessary import of EmailMessage.
Usually we don't bother with unused imports, but this helps with
reasoning about where we use the Django mail APIs in Zulip.
2018-07-12 12:32:27 +05:30
Joshua Pan 533eccd655 models: Create delivery_email field in userprofile.
This commit creates a new field called delivery_email. For now, it is
exactly the same as email upon user profile creation and should stay
that way even when email is changed, and is used only for sending
outgoing email from Zulip.

The purpose of this field is to support an upcoming option where the
existing `email` field in Zulip becomes effectively the user's
"display email" address, as part of making it possible for users
actual email addresses (that can receive email, stored in the
delivery_email field) to not be available to other non-administrator
users in the organization.

Because the `email` field is used in numerous places in display code,
in the API, and in database queries, the shortest path to implementing
this "private email" feature is to keep "email" as-is in those parts
of the codebase, and just set the existing "email" ("display email")
model field to be something generated like
"username@zulip.example.com" for display purposes.

Eventually, we'll want to do further refactoring, either in the form
of having both `display_email` and `delivery_email` as fields, or
renaming "email" to "username".
2018-07-12 12:30:20 +05:30
Joshua Pan 4b3fb746ea Remove incorrect setting of user_profile.email.
user_profile.save() is never called, so this line of
code is actually useless.
2018-07-12 12:23:46 +05:30
Eeshan Garg 3eaf00444a markdown: Render nested multi-line code blocks correctly.
This commit adds a Markdown tree-processor extension that renders
multi-line code blocks that are nested inside lists with the
formatting. Note that the code block could be nested inside multiple
list levels and would still get rendered correctly.

Tim: This fixes the need for unpleasant workarounds like
f5bfa4e793 and makes nested code blocks
in our documentation look exactly how users would expect them to.
2018-07-12 12:22:04 +05:30
Yago González a3d42d9901 test-api: Pass non-admin client to the test helpers. 2018-07-11 23:54:00 +05:30
Rhea Parekh ecc5b75500 tests: Refactor test_import_realm for clarity. 2018-07-11 23:37:06 +05:30
Yago González e9cd0ad3d6 reactions: Allow using emoji_name for removing reactions.
Given that we allow adding emoji reactions by only using the
emoji_name, we should offer the same possibility for removing
reactions to make the experience for API clients not require looking
up emoji codes.

Since this is an additional optional parameter, this also preserves
backward compatibility.
2018-07-11 19:41:43 +05:30
Yago González 9244a6740e reactions: Make the emoji_code optional for adding a rection.
Complete, correct implementations of Zulip's emoji reactions API need
to send both emoji_code and emoji_name in order to add a reaction;
this is important for corner cases around clicking on a reaction in a
message that was first reacted to a year ago, when the emoji
name->code mappings have changed for the given code point in the
intervening time.

However, for folks building tools using the Zulip API, that corner
case is not particularly common; as a result, it makes sense to offer
an interface that allows adding a reaction by only specifying the
emoji name.

This is why the only field that needs to be required is emoji_name,
which can now be mapped to a single emoji. Both fields will be
necessary when "voting" an old reaction, but since we stil allow
specifying the two of them, these changes offer retrocompatibility.
2018-07-11 19:34:48 +05:30
Aditya Bansal 9b485f3ef4 auth: Fix bug with subdomains and GitHub auth causing apparent logouts.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.

If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in.  The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!

We fix this by moving python-social-auth to a separate subdomain.

Fixes: #9847.
2018-07-10 16:49:00 +05:30
Rhea Parekh 4bbccd8287 import: import RealmAuditLog when 'zerver_realmauditlog` is missing.
* If `zerver_realmauditlog` is present in the exported data,
  `RealmAuditLog` would be imported normally.

* If it is not present, `create_subscription_events`
  function in would create the `subscription_created`
  events for RealmAuditLog. The reason this function
  is in `import_realm` module and not in the individual
  export tool scripts (like Slack) is because this
  function would be common for all export tools.

This fixes #9846 for users who have not already done an import of
their organization from Slack.

Fixes #9846.
2018-07-10 16:00:19 +05:30
Rhea Parekh 70b4794816 import: import RealmAuditLog. 2018-07-10 15:53:15 +05:30
Rhea Parekh c182a0c7a0 export: export RealmAuditLog. 2018-07-10 15:53:15 +05:30
Vishnu Ks c7cb0c6aa0 models: Add USER_SOFT_DEACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 547b5675c6 models: Add USER_SOFT_ACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks a0da184d50 models: Add SUBSCRIPTION_ACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks abb218ebab models: Add SUBSCRIPTION_DEACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks ff4c1ca2c1 models: Add SUBSCRIPTION_CREATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 20fae065f6 models: Add BOT_OWNER_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 9670e319cc models: Add REALM_REACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks d81cf2268a models: Add REALM_DEACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks d5b6f032d2 models: Add USER_API_KEY_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 0d69fc482b models: Add USER_TOS_VERSION_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 83ff688bb6 models: Add USER_EMAIL_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 75ed3f92e2 models: Add USER_FULL_NAME_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks c8aac0fbb9 models: Add USER_CHANGE_AVATAR_SOURCE event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 4c73221108 models: Add USER_CHANGE_PASSWORD event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 201b99a6f8 models: Add USER_REACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks d0b89cbb44 models: Add USER_DEACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks ce3fffdbb2 models: Add USER_ACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 2c8effe9fe models: Add USER_CREATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Rhea Parekh fe4cad15a4 widgets: Add todo widget. 2018-07-10 11:18:05 +05:30
Yashashvi Dave 649fccde6b zerver/lib/events.py: Fix custom field values not updated in apply_events.
Custom profile field value are stored in different structure compare to
other profile fields in events, so generic way to update fields wasn't
updating custom profile fields in `apply_event` function.

Fix this by adding check for custom fields in `apply_event`.

This also adds the appropriate test_events test to verify this code path.

Fixes part of #9875.
2018-07-10 10:30:23 +05:30
Aditya Bansal 0c5dadf320 actions.py: Extract topic history list generation as a function.
We extract out the logic for generating a list of all historical
topics for a given stream as a separate function.  This avoids code
duplication when we add the similar code path for grabbing all topics
for web public streams.
2018-07-10 10:22:07 +05:30
Shubham Dhama 49deb5acd3 bugdown: Move global variables to arguments.py.
This has two advantages;

* We can split bugdown/__init__.py into several modules, and each
  module can access these arguments by importing these

* We get rid of the super-ugly `global db_data` construct, replacing
  it with a only slightly ugly monkey-ish patching of the
  `zerver.lib.bugdown.arguments` module, which is at least
  considerably more clear on reading as to what it's purpose is.
2018-07-09 15:45:45 +05:30
Shubham Dhama b9a8c4df8c bugdown: Move BugdownRenderingException to exceptions.py. 2018-07-09 15:45:45 +05:30
Rishi Gupta b5753d0ddc billing: Add initial support for seat based plans.
The main remaining todo for correctly populating
RealmAuditLog.requires_billing_update is supporting the de-seating (and
corresponding re-seating) that happens after being offline for two weeks.
2018-07-09 14:33:08 +05:30
Rishi Gupta 16334a1ba7 billing: Update billing system. 2018-07-09 14:33:07 +05:30
Aditya Bansal 872e8c1d7b test_fixtures: Fix bug with run_db_migrations for test platform.
In this commit we are fixing a kinda serious un-noticed bug with
the way run_db_migrations worked for test db.
Basically run_db_migrations runs new migrations on db (dev or test).
When we talk about the dev platform this process is straight forward.
We have a single DB zulip which was once created and now has some data.
Introduction of new migration causes a schema change or does something
else but bottom line being we just migrate the zulip DB and stuff works
fine.

Now coming to zulip test db (zulip_test) situation is a bit complex
in comparision to dev db. Basically this is because we make use of
what we call zulip_test_template to make test fixture restoration
after tests run fast. Now before we introduced the performance
optimisation of just doing migrations when possible, introduction of
a migration would ideally result in provisioning do a full rebuild of
the test database. When that used to happen sequence of events used to
be something like this:
* Create a zulip_test db from zulip_test_base template (An absolute
basic schema holding)
* Migrate and populate the zulip_test db.
* Create/Re-create zulip_test_template from the latest zulip_test.

Now after we introduced just do migrations instead of full db rebuild
when possible, what used to happen was that zulip_test db got
successfully migrated but when test suites would run they would try to
create zulip_test from zulip_test_template (so that individual tests
don't affect each other on db level).
This is where the problem resides; zulip_test_template wasn't migrated
and we just scrapped zulip_test and re-created it using
zulip_test_template as a template and hence zulip_test will not hold the
latest schema.

This is what we fix in this commit.
2018-07-09 14:17:53 +05:30
Armaan Ahluwalia 54d3d8e8b3 webpack: Transition app.js to be compiled by webpack.
This commit moves all files previously under the 'app' bundle in
the Django pipeline to being compiled by webpack under the 'app'
entry point. In the process, it moves assets under the app entry
to a file called app.js that consumes all relevant css and js files.

This commit also edits the webpack config to be able to expose certain
variables for third party libraries that are currently required by
some modules. This is bad coding form and should be refactored to
requiring whatever dependencies a module may have; we're just
deferring that to the future to simplify the series of transitions we
need to do here. The variable exposure is done using expose-loader in
webpack.

The app/index.html template is edited to override the newly introduced
'commonjs' block in the base template. This is done as a temporary
measure so as not to disrupt other pages on the app during the transition.

It also fixes the value of the 'this' context that was being inferred
as window by third party libraries. This is done using imports-loader
in the webpack config.  This is also messy and probably isn't how we
want things to work long term.
2018-07-05 11:03:08 +02:00
Raymond Akornor 5ef6689a69 minified_js: Clean up unused imports. 2018-07-04 17:38:55 +02:00
Yago González fa2daf0044 api docs: Migrate POST /users/me/subscriptions to OpenAPI. 2018-07-03 20:15:34 +02:00
Yago González f3277500de api docs: Add missing space in exception's message. 2018-07-03 20:11:05 +02:00
Yago González 2d1adc8d99 api docs: Allow validation against schemas for any response. 2018-07-03 20:11:05 +02:00
Tim Abbott 5a99118b3e auth: Restore a minimal SocialAuthMixin.
We need to do a small monkey-patching of python-social-auth to ensure
that it doesn't 500 the request when a user does something funny in
their browser (e.g. using the back button in the auth flow) that is
fundamentally a user error, not a server error.

This was present in the pre-rewrite version of our Social auth
codebase, without clear documentation; I've fixed the explanation
part here.

It's perhaps worth investigating with the core social auth team
whether there's a better way to do this.
2018-07-03 18:53:59 +02:00
Tim Abbott c9b0c0add4 github: Refactor email extraction to use the full emails data set.
It's possible to make GitHub social authentication support letting the
user pick which of their verified email addresses to pick, using the
python-social-auth pipeline feature.  We need to add an additional
screen to let the user pick, so we're not adding support for that now,
but this at least migrates this to use the data set of all emails that
have been verified as associated with the user's GitHub account (and
we just assume the user wants their primary email).

This also fixes the inability for very old GitHub accounts (where the
`email` field in the details might be a string the user wanted on
their GitHub profile page) to using GitHub auth to login.

Fixes #9127.
2018-07-03 18:35:29 +02:00
Anders Kaseorg d8ba378050 test_slack_importer: Remove backslashes wrongly copied from JSON data
https://github.com/houstondatavis/slack-export/blob/master/users.json

JSON or JavaScript decodes "\/" to / (and some encoders always write
"\/" to avoid accidentally creating a </script> tag), while Python
assumes "\/" is a typo for "\\/" and decodes it to \/.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Rhea Parekh fdef1d8d91 zcommand: Raise error when command doesn't begin with a slash. 2018-07-01 20:32:36 -04:00
Rhea Parekh b22d266667 widgets: Add question for poll widget in the message itself.
Use the command '/poll question?', to start a question.
2018-07-01 19:56:19 -04:00
Lenny Jagielski b9393387e1 integrations: Fix heroku integration logging.
Now, every line in the commit log is properly quoted, but using a
Zulip markdown block quote (rather than just a single `>`).

Fixes: #9792
2018-07-01 12:43:31 -07:00
Eeshan Garg bf175f6331 webhooks/github: Add support for PR review requests.
Fixes: #9732.
2018-07-01 12:40:45 -07:00
Eeshan Garg 04ed123214 webhooks/github: Be more explicit about unsupported PR events.
This was technically a bug. For events that aren't unsupported
intentionally, the control should fall to the line that raises
UnknownWebhookEventType, and shouldn't be handled by anything else.

The events that are intentionally unsupported should be handled
more explicitly.
2018-07-01 12:40:45 -07:00
Yago González d151f4a234 messages: Allow fetching unedited messages' history.
When GETting an unedited message's edit history, the server wasn't able
to reply properly and produced a 500 error.

Now when that happens, we return a message history that only contains
the original message.
2018-07-01 08:16:59 -07:00
Rhea Parekh 838ab2fce5 export: Add variable MESSAGE_BATCH_CHUNK_SIZE in export.py.
Also use this variable in slack_data_to_zulip_data.
2018-07-01 07:08:13 -07:00
Rhea Parekh 6b7b6b38ad slack import: Write messages batch-wise.
Messages can be bulky, and storing them in a single
data structure can cause a memory error.

In this commit, the messages are written to a file
batch-wise, thus avoiding the memory error.
2018-07-01 07:08:13 -07:00
Rhea Parekh 7f6c174099 slack import: Add 'id_list' field in channel_message_to_zerver_message.
The id_list would help to store the associated max ID
state between subsequent calls, which will help
in batch-wise processing of the messages.
2018-07-01 07:08:13 -07:00
Rhea Parekh af20ef4789 slack import: Save messages within convert_slack_workspace_messages.
Previously, the messages where being stored in a output file from
outside the function 'convert_slack_workspace_messages', but
now we store it from the inside the mentioned function.
This will help in processing and saving the messages batch-wise
so as to avoid a memory error.

Reactions are returned separately from 'convert_slack_workspace_messages'
rather than 'message_json'.

Also updated test for 'convert_slack_workspace_messages' and an additional
test for reactions is added.
2018-07-01 07:08:13 -07:00
Rhea Parekh 2f88ca7446 slack import: Import skype and phone data of users. 2018-07-01 07:05:40 -07:00
Eeshan Garg 061e760d1e webhooks: Add Clubhouse integration. 2018-07-01 04:01:17 -07:00
Tim Abbott 5f7691b74e upload: Remove unnecessary use of has_request_variables.
All the parameters for this function are parsed in urls.py.
2018-07-01 01:47:03 -07:00
Eeshan Garg e1e4d523aa Replace remaining user-facing occurences of the term "invite-only".
Fixes #9611.
2018-06-27 12:34:29 -07:00
Steve Howell a2a695dfa7 test flake: Prevent flaky test with less common id.
This fixes a test flake introduced here:

    317a2fff2a

We need a higher bogus bot owner id to prevent
flakes where our userid sequence gets to 100.  (Tests
aren't completely deterministic in what data you
use, since sequences don't get rolled back when
you roll back transactions.)
2018-06-26 16:56:24 -04:00
Rishi Gupta 2de9029a69 stream settings: Add tests for weekly stream traffic calculation. 2018-06-23 12:48:15 -07:00
Rishi Gupta dc7cfd3eb7 stream settings: Fix stream_age calculation in weekly traffic calculation. 2018-06-23 12:48:15 -07:00
Rishi Gupta 3054b175c9 stream setting: Fix mypy error in get_average_weekly_stream_traffic.
Not sure why mypy wasn't catching this.
2018-06-23 12:48:15 -07:00
Rishi Gupta 15b236a21f stream settings: Show non-zero stream traffic if traffic is greater than 0.
An estimated traffic of 0 suggests a stream is dead, and has pretty
different semantics from any non-zero value. So we should round up any
number between 0 and 1 to 1.
2018-06-23 12:48:15 -07:00
Rishi Gupta 5eccabc3c5 stream settings: Set average_weekly_traffic to -1 if stream is new.
We don't ever use this value, but it's confusing to have the incorrect
calculation in the code.

Ideally we would set this to "None", but I don't know the code well enough
to be confident nothing would break.
2018-06-23 12:48:15 -07:00
Rishi Gupta f63bcb99d7 stream settings: Refactor out STREAM_TRAFFIC_CALCULATION_MIN_AGE_DAYS. 2018-06-23 12:48:15 -07:00
Rishi Gupta ccf5f1ac89 stream settings: Fix weekly stream traffic computation.
Previously, if stream_age were 13, this number would be off by almost a
factor of 2.
2018-06-23 12:48:15 -07:00
Yashashvi Dave 317a2fff2a bots settings: Improve API for editing bot owners to refer to users by ID.
Fixes #9502
2018-06-23 12:47:50 -07:00
Vishnu Ks 6e6e2e03da management: Use tokenized no-reply address in send_test_email. 2018-06-23 12:03:30 -07:00
Vishnu Ks 1b179ca530 signup: Prevent users from signing up with email containing +. 2018-06-23 12:03:30 -07:00
Vishnu Ks a88e44e254 registration: Use tokenized noreply address in email change. 2018-06-23 12:03:30 -07:00
Vishnu Ks e34fcf982f registration: Use tokenized noreply address in user invite. 2018-06-23 12:03:30 -07:00
Vishnu Ks 994e1a2154 registration: Use tokenized noreply address in password reset. 2018-06-23 12:03:30 -07:00
Vishnu Ks 2b1424e51a registration: Use tokenized noreply address in signup confirmation. 2018-06-23 12:03:30 -07:00
Vishnu Ks 74351fb250 send_email: Add function to generate tokenized noreply in FromAddress. 2018-06-23 12:03:30 -07:00
Yago González 51f107f5bc api docs: Fix indentation. 2018-06-23 12:42:11 +02:00
Yago González 8b420da653 api docs: Remove unnecessary imports. 2018-06-23 12:42:11 +02:00
Yago González 8e50564717 api docs: Document the /users/me/<stream_id>/topics endpoint. 2018-06-23 12:42:11 +02:00
Raymond Akornor af5de189d8 tests_users: Use assert_json_success instead of assertTrue. 2018-06-22 09:06:43 -04:00
Marco Burstein 6dad6f3dae emoji: Add some edge-casey ordering tests for emoji.
Add 3 new Markdown emoji tests for newlines, emphasis, and links. The
goal of these tests is to ensure that Markdown operations concerning
emoji are preformed in proper order, with emoji being added correctly
based on other Markdown operations.

See suggestion here: https://git.io/flF5W.
2018-06-22 08:59:08 -04:00
Rhea Parekh d066c11b47 zcommand: Strip the slash in the backend.
The slash in command is stripped in the backend,
rather than in the client to make the client code
cleaner.
This would make client code cleaner in the slash
commands which include parameters.
2018-06-20 16:49:29 -04:00
Rhea Parekh f4bbd4f1a3 tests: Use Set instead of List in import tests. 2018-06-20 10:22:34 -07:00
Marco Burstein 02489e5fb8 emoji: Fix a bug where emoticon translation is not done after a newline.
This bug is caused by the conversion of newlines to `<br>` statements,
since `>` is not allowed as a character around an emoticon during
translation.

Also, add a new test case for preventing this bug from occurring in the
future.

Fix #9763.
2018-06-20 08:09:18 -07:00
Rishi Gupta 4f8c44a019 notifications: Simplify notification messages for new subscriptions.
We're adding more stream types, e.g. splitting private streams into
with/without shared history, adding publicly-archived streams, adding
announce-only streams, etc. So maintaining this text is going to get more
complicated over time.

Also, the right place to explain this stuff is in the stream header, or near
the z-in-a-circle.

This commit also adds translation tags to the messages.
2018-06-20 07:53:10 -04:00
Vishnu Ks 403f254557 signup: Create get_accounts_for_email function. 2018-06-19 11:25:23 -07:00
Shubham Dhama 8e032376f9 guest: Restrict guest access to user group creation and updation. 2018-06-19 11:04:14 -07:00
Rhea Parekh 200c381c91 tests: Add tests for importing files from S3 backend. 2018-06-19 10:50:11 -07:00
neiljp (Neil Pilgrim) 88a70c5f5d mypy: test_embedded_bot_system.py: add asserts & remove from mypy.ini. 2018-06-19 10:48:38 -07:00
neiljp (Neil Pilgrim) e4821875f7 mypy: Improve typing of oembed data, to Dict[str, Any]. 2018-06-19 10:48:38 -07:00
neiljp (Neil Pilgrim) b5aa705137 mypy: test_link_embed.py: add assert & remove from mypy.ini. 2018-06-19 10:48:38 -07:00
Rhea Parekh d1ba6bae03 import: 'processing_emojis' and 'processing_avatars' should now be True together.
Raise an exception when the fields
'processing_emojis' and 'processing_avatars' are
True at the same time. Also add test for the same.
2018-06-18 23:06:09 +05:30
Rhea Parekh a2a74d9271 export: The records.json IDs should be integer.
In records the IDs like the realm_id and user_profile_id
of 'records.json' should be integers. This was missing in the
S3 backend and this commit fixes that.

Added tests for this as well.
2018-06-18 23:06:09 +05:30
Rhea Parekh 4d21f7f747 import: 'attachment_path' should be saved with the 's3_path' of the record.
For the S3 backend uploads, 'attachment_path' should be
saved with the 's3_path' of the record, as the original
'path' is changed while exporting files from s3. (See
function 'export_files_from_s3' in export.py for reference.)
2018-06-18 23:06:01 +05:30
Rhea Parekh 0730087111 import: Add elif condition for 'processing_emojis' in 'import_uploads_s3'.
'processing_emojis' should have an 'elif' condition here as
we want the function to work for avatars, emojis or uploads
one at a time.
2018-06-18 23:04:18 +05:30
Rhea Parekh f2b5f5a8f9 import: Fix processing_avatars bug in 'import_uploads_s3'.
All the avatars should be processed later on to run the
'ensure_medium_avatar_image' function. This is similar to
'import_uploads_local'.
2018-06-18 22:37:34 +05:30
Rhea Parekh f66ca9a5c3 import: Pass 'processing_emojis' in 'import_uploads_s3'.
'import_uploads_s3' should be passed with the parameter
'processing_emojis' from 'import_uploads'.
2018-06-18 22:35:36 +05:30
Rhea Parekh b2e971b9b1 tests: Add tests for the export file's records.
For the emojis, In 'records.json', the record should contain
the attribute 'file_name', which was missing in the S3 backend.
This commit adds this attribute, as well as tests for the
records of uploads, avatars and emojis in both local and S3 backend.
2018-06-18 09:19:24 -07:00
Rhea Parekh 5cbad76445 tests: Add tests for importing files from local. 2018-06-18 09:19:24 -07:00
Rhea Parekh edb506a605 tests: Add a function to setup export of files.
Add the function '_setup_export_files' to setup the files to
be exported.
2018-06-18 09:19:24 -07:00
Rhea Parekh f735fae777 tests: Add tests for a basic import logic. 2018-06-18 09:19:24 -07:00
Rhea Parekh 0e8f0e2393 tests: Rename 'message.json' to 'messages-000001.json' in export tests.
The export script produces messages in the format 'messages-%06d.json'.
Hence we rename it to 'messages-000001.json'.
2018-06-18 09:19:24 -07:00
Yashashvi Dave 8909cb1d15 custom fields: Allow list of users in user type of custom fields.
Allow user to add more than one user-value in user type of custom
fields.

Tweaked by tabbott to improve the models.py code and type annotations.
2018-06-16 09:37:49 -07:00
Eeshan Garg 943a77105b integrations/irc: Capitalize the integration name properly.
It is IRC, not Irc, which is what the default code in Integration
renders.
2018-06-16 08:52:49 -07:00
Vishnu Ks 4da61e2a17 signup: Copy enter_sends value when importing settings. 2018-06-16 08:50:49 -07:00
Vishnu Ks b639778d0f signup: Copy hotspots when importing settings. 2018-06-16 08:50:49 -07:00
Rhea Parekh d0bc8d0736 zcommands: Add zcommand module and separate test module.
Move the zcommands from '/views/messages.py' to
'/lib/zcommand'.

Also, move the zcommand tests from '/tests/test_messages.py'
to '/tests/test_zcommand'.
2018-06-16 10:32:42 -04:00
Shubham Dhama b0bfb6b153 streams: Hide create stream UI from guest users. 2018-06-16 06:33:14 -07:00
Shubham Dhama d40c2bb166 emoji: Make uploading new realm emoji inaccessible for guest users. 2018-06-16 06:33:14 -07:00
Shubham Dhama 596ce5b60f invite: Make inviting new users inaccessible for guest users. 2018-06-16 06:33:13 -07:00
Tim Abbott 425b4a3866 invite: Use require_realm_admin decorator. 2018-06-16 06:33:13 -07:00
Neil Pilgrim ba55d22fdb mypy: Improve MessageOutput typing in export.py.
See the comments above for why this is the correct list of options.
2018-06-14 15:22:56 -07:00
Eeshan Garg 05b752bd3f webhooks/trello: Mention that Trello script can be run from anywhere.
This is one of those weird webhooks where the
download-python-bindings.md macro doesn't work, because the user
only needs the bindings to run the one-time Trello script to register
the webhook and that script can be run from anywhere and doesn't need
to be "hosted" anywhere.
2018-06-11 07:46:16 -04:00
Tim Abbott 93b5e0df61 custom profile fields: Extract a separate various for whether enabled.
This will make it possible to do this on a per-organization basis.
2018-06-07 11:35:04 -07:00
Roman Godov 0b9d4fbb06 auth: Added time conversion in new login emails.
This commit adds code for date and time conversion in
"new login" emails according to user's timezone.

Fixes #9336.
2018-06-07 08:40:22 -07:00
Yashashvi Dave f7f039e772 custom fields: Add endpoint to delete value of custom profile data.
Add delete endpoint at `users/me/profile_data` to delete value of
custom profile data.

Fixes #9669
2018-06-07 08:35:04 -07:00
Adam Birds 5855fe3e53 integrations: Add Zabbix webhook.
With a few changes to the documentations by Eeshan Garg.
2018-06-07 08:31:24 -07:00
Eeshan Garg 4446b76427 pypi: Upgrade to release 0.5.0. 2018-06-07 08:31:21 -07:00
MaTaM673 d27c230982 webhooks/teamcity: Include branch name in build notifications. 2018-06-07 08:15:30 -07:00
Tim Abbott 4481c74a71 GitHubAuthBackendTest: Remove token_data_dict argument.
This was always the same, and there's not much reason to customize it.
2018-06-06 15:22:52 -07:00
Aditya Bansal 1205e02c64 provisioning: Don't rebuild DB if running migrations is sufficient.
This results in a significant optimization in the performance of
re-provisioning Zulip if all that you're doing is rebasing onto a
newer version of master (which just adds new migrations).

The change carries some risk of generating unpleasant-to-debug
situations, because if we merge a buggy migration and then later fix
it, some clients may not have a properly migrated database (and also,
this changes how populate_db commutes with migrations).  But it seems
worth it, given how much time is currently wasted by not having this.

Fixes: #9512.
2018-06-06 13:43:59 -07:00
Aditya Bansal 65dc80fe9d test_fixtures: Add run_generate_fixtures_if_required function.
In this commit we are adding run_generate_fixtures_if_required,
a new function which is meant to de-duplicate a bit of code
between test-server and test-backend which is essentially
responsible for rebuilding the test database if that was required.
2018-06-06 13:37:35 -07:00
Aditya Bansal f7c11d1747 test_fixtures: Refactor to have template_database_status API.
In this commit we are essentially just refactoring the function
is_template_database_current to be called template_database_status
and adjusting the return values accordingly.
This is essentially a preparatory commit for the upcoming commits
which will essentially enable us to not throw away entire DB and
rebuild from scratch if only running migrations could do the job.
2018-06-06 13:37:35 -07:00
Vishnu Ks 53237d39aa registration: Allow users to import profile picture. 2018-06-06 13:29:42 -07:00
Vishnu Ks ca87cf4c97 upload: Seperate writing avatars into s3 as function. 2018-06-06 13:29:42 -07:00
Vishnu Ks 7960785261 upload: Separate writing avatars as local files into function. 2018-06-06 13:29:42 -07:00
Tim Abbott 7d250cb2f9 attachment: Fix handling of stream history.
This fixes two issues:

* Our guest users feature gave guest users access to public stream
  attachments even if they couldn't access the public stream.

* After a user joins a private stream with our new shared history
  feature, they couldn't see images uploaded before they joined.

The tests need to check for a few types of issues:
* The actual access control permissions.
* How many database queries are used in the various
  cases for that second model, especially with multiple messages
  referencing an attachment.  This function gets called a lot, and we
  want to keep it fast.

Fixes #9372.
2018-06-06 09:40:22 -07:00
Tim Abbott 541ccfeb7f validate_attachment_request: Simplify execution flow.
This makes the code more readable, by narrowing the try/except block
and only querying `messages` when we're going to use it.
2018-06-06 09:13:51 -07:00
Tim Abbott 35c4a9f1d2 auth: Rewrite our social auth integration to use pipeline.
This new implementation model is a lot cleaner and should extend
better to the non-oauth backend supported by python-social-auth (since
we're not relying on monkey-patching `do_auth` in the OAuth backend
base class).
2018-06-05 23:24:48 -07:00
Yashashvi Dave 5145b24635 users: Replace duplication with generic func to validate user id.
This adds a common function `access_user_by_id` to access user id
within same realm, complete with a full suite of unit tests.

Tweaked by tabbott to make the test much more readable.
2018-06-05 11:13:13 -07:00
Umair Khan 9b698dec08 2FA: Use patch.object for monkey patching.
Ref #9521
2018-06-05 09:25:40 -07:00
Tim Abbott 238ba2d486 auth: Move some simple GitHub tests to the long-term test class. 2018-06-04 23:57:09 -07:00
Tim Abbott 869d75b02f AuthBackendTest: Fix typos in error message checks.
Previously, these checks did nothing.
2018-06-04 23:45:13 -07:00
Raymond Akornor 0335da7e05 python: Use python3 style super() in 2FA code paths. 2018-06-04 20:55:42 -07:00
Shubham Dhama 01555e8772 streams: Handle guest user ids for stream settings changes' events. 2018-06-04 11:35:37 -07:00
Steve Howell a42492d0ac Fix very recent build error w/url coverage for casper.
Example:
    http://localhost:9991/casper/casper-failure1.png
2018-06-04 12:37:10 -04:00
Tim Abbott ec90322e5d bots: Fix behavior for bots mentioned in PMs.
We've for a long time had the behavior that a bot mentioned in a
stream message receives the notification, regardless of whether the
bot was actually subscribed to the stream.

Apparently, this behavior also triggered if you mentioned a bot in a
private message (i.e. the bot would be delievered the private message
and would probably respond unhelpfully in a new group private message
thread with the PMs original recipients plus the bot).

The fix for this bug is simple: To exclude this feature for private
messages.
2018-06-03 22:29:15 -07:00
Tim Abbott ec2eb0edba storage: Don't double-minify webpack bundles.
What was happening before is that we built the webpack bundles in
tools/minify-js with nicely hashed filenames, and then `manage.py
collectstatic` was extending these filenames with a second hash
through the use of storage.

Removing the first one didn't seem ideal, but would probably have
worked, but seems confusing for people only familiar with webpack
(ideally, we want the Django toolchain piece to be increasingly
invisible as we replace it).

And we can't exclude the webpack bundles from being processed by
storage, since we need these bundles to be included in the manifest.
So, instead, we set the hash function to be a no-op for the bundle
files.

Fixes significant portions #5971.

More work is required to deal with versioning for some of the
image/font assets.
2018-06-03 16:49:59 -07:00
Tim Abbott 714ad67db1 storage: Move library code into zerver/lib.
There's no particularly good reason for this to have been a top-level
file.
2018-06-03 16:02:02 -07:00
Rishi Gupta 182215d125 portico: Update text of why-zulip.
With edits from tabbott.
2018-06-03 09:37:00 -07:00
Shubham Dhama 4483e33102 digest: Make newly registered users data inaccessible to guest users.
The new can_access_all_realm_members function is meant to act as a
base function for guest users and Zephyr realm users regarding the
accessibility of the information of other users in the realm.
2018-06-03 09:30:59 -07:00
Shubham Padia 8883107a7f streams: Allow cross-realm bots to post to announcement_only streams.
This fixes an issue where if you make #announce (the default
announcement stream) announce-only, then creating a new stream will
throw an exception (because notification-bot can't send there).

Fixes #9636.
2018-06-02 10:36:29 -07:00
Steve Howell 47b4dd6bdb slash commands: Refine /day and /night.
These two slash commands now use zcommand to talk to
the server, so we have no Message overhead, and if you're
on a stream, you no longer spam people by accident.

The commands now also give reasonable messages
if you are already in the mode you ask for.

It should be noted that by moving these commands out of
widget.py, they are no longer behind the ALLOW_SUB_MESSAGES
setting guard.
2018-06-02 09:40:12 -07:00
Steve Howell 4b2e8b83c4 slash commands: Add /ping command (via zcommand).
This adds a /ping command that will be useful for users
to see what the round trip to the Zulip server is (including
only a tiny bit of actual server time to basically give a
200).

It also introduce the "/zcommand" endpoint and zcommand.js
module.
2018-06-02 09:40:12 -07:00
Rhea Parekh f8e53511ea slack import: Add support for bold-italics formatting.
Fixes #8927
2018-06-02 09:01:55 -07:00
Tim Abbott 18702b0d71 actions: Document validate_user_access_to_subscribers_helper.
This function is important, carefully optimized, and hard to read, so
it deserves a long docstring.
2018-06-02 08:46:58 -07:00
Shubham Dhama 8e2337509d streams: Hide public streams & subscribers from guest users.
The main change here is making can_access_public_streams
2018-06-02 08:39:39 -07:00
Shubham Dhama ca02143074 test_subs: Fix typos in test_previously_subscribed_private_streams. 2018-06-02 08:06:45 -07:00
Tim Abbott 751c602a0b update-prod-static: Don't copy/minify templates and styles.
This is a performance optimization: Rather than copying these files
into the `prod-static` directory and then deleting them, we just don't
copy them over in the first place.

For styles, it might have once been the case that this did something,
but we've moved them all to being managed by webpack some time ago.

For the js directory, I think it was never useful to copy and then
delete them; these files were always compiled via tools/minify-js,
and the raw JS files weren't needed, anyway.
2018-06-01 16:39:35 -07:00
Tim Abbott 2a52b0c1e8 compilemessages: Use json.dump with sort_keys option.
This should help make these files stable over time.  Mostly useful for
diffing release tarballs.
2018-06-01 16:39:03 -07:00
Eeshan Garg 52c3c00ea0 test-api: Test against generic fixtures.
In a few commits before this one, we just added de-duplicated
generic fixtures that apply to multiple API tests. The tests
needed to be modified to accommodate that change.
2018-06-01 14:06:26 -07:00
Yago González 9f98002b76 tests: Allow testing our REST API against the OpenAPI docs. 2018-06-01 12:03:31 -07:00
Tim Abbott c37fe998d2 templates: Make zulip_default_context available in API endpoints.
This is mostly useful for advertising support_email, but some other
fields may by helpful over time.
2018-06-01 11:32:18 -07:00
Tim Abbott 0d84eb0a8b mypy: Clean zerver/views/users.py for strict-optional. 2018-06-01 08:48:17 -07:00
Tim Abbott 0a0ae5e703 users: Move add_service to zerver.lib.users so it is reusable. 2018-06-01 08:48:17 -07:00
Eeshan Garg 10523813fd webhooks/pagerduty: Update docs to conform to style guide.
A few things have changed on PagerDuty's end, so I thoroughly tested
this webhook with a local ngrok instance. Also, added a message
screenshot!
2018-05-31 23:48:42 -04:00
Eeshan Garg 1df6c22df0 webhooks/pagerduty: Fix small capitalization error in docs.
It is PagerDuty, not Pagerduty!
2018-05-31 23:42:07 -04:00
Tim Abbott bab69a325f registration: Narrow try/exception block for LDAP usernames.
This just makes the code a bit more clear about where we expect that
exception to come from.
2018-05-31 14:04:47 -07:00
Tim Abbott 059d6672ba query_ldap: Print the email address for configuration (C).
This should make it easier to debug whether LDAP configuration (C) is
working correctly.
2018-05-31 13:47:26 -07:00
Sampriti Panda 46711a43f0 pgroonga: Upgrade to PGroonga 2 API. 2018-05-31 13:00:34 -07:00
Sampriti Panda e95f972e24 migrate: Add do_batch_update method for running batch updates.
* Refactor pgroonga_0002 migration to use new method.
2018-05-31 13:00:34 -07:00
Tim Abbott b9b81cf658 export: Rename ALL_ZERVER_TABLES to ALL_ZULIP_TABLES.
They don't all start with zerver, now :).
2018-05-31 10:47:27 -07:00
Tim Abbott 42aea68df3 export: Automate validation of ALL_ZERVER_TABLES.
This should help make it explicit whenever we add a new table to Zulip
that we need to correctly categorize it for whether it will be
included in the data export, or not.
2018-05-31 10:47:27 -07:00
Tim Abbott 328136344a import: Fix typo in zerver_customprofilefieldvalue table name.
Apparently, we were doing this slightly wrong.
2018-05-31 10:47:27 -07:00
Shubham Padia cd1233d3f7 upload: Add test jpg image with no exif data. 2018-05-31 08:08:59 -07:00
Tim Abbott 5967de9177 test_auth_backends: Move GitHub signup tests to new suite.
This eliminates a lot of duplicated, mocking-heavy code.
2018-05-30 18:24:27 -07:00
Tim Abbott 7d8d6f1978 test_auth_backends: Move GitHub deactivated test to new suite. 2018-05-30 18:24:27 -07:00
Tim Abbott 29492325a8 test_auth_backends: Move github_auth_enabled test to new suite.
This is step 1 of a migration to eventually delete the legacy test suite.
2018-05-30 18:24:27 -07:00
Steve Howell 554883e039 submessages: Add submessage_id to submessage event.
It's useful to have the submessage_id so that widgets
can in theory sort all the events (and not have to rely
on them arriving in sequence).
2018-05-30 14:47:36 -07:00
Shubham Padia 214ce1ccca streams: Add create and edit ui for is_announcement_only.
The user can now specify the value while creating a stream.
An admin can later change it via `Change stream permissions`
modal. Add is_announcement_only to subscription type text.
2018-05-30 14:22:15 -07:00
Shubham Padia bb8ad15fa9 streams: Set is_announcement_only when creating streams.
Adds is_announcement_only to create_stream_if_needed and
add_subscriptions_backend.
2018-05-30 14:17:33 -07:00
Steve Howell bfc13ecea4 submessages: Just send content for submessage events.
For some reason in my original version I was sending both
content and data to the client for submessage events,
where data === JSON.parse(content).  There's no reason
to not just let the client parse it, since the client
already does it for data that comes on the original
message, and since we might eventually have non-JSON
payloads.

The server still continues to validate that the payload
is JSON, and the client will blueslip if the server
regressses and sends bad JSON for some reason.
2018-05-30 13:53:11 -07:00
Rhea Parekh e2f2ea7248 test_export: Add tests for 'do_export_user'. 2018-05-30 10:28:10 -07:00
Rhea Parekh 74cc07bb2f test_export: Add tests for export of files.
Thanks @Zachary-Jackson for some preliminary work
on this.
2018-05-30 10:28:10 -07:00
Tim Abbott f8e8d1fb27 upload: Suppress coverage warnings for now. 2018-05-30 09:31:47 -07:00
Tim Abbott bbcdf8ca99 upload: Fix logic for orientation when uploading avatars.
The previous logic was broken for an avatar without exif data.
2018-05-30 09:22:36 -07:00
Tim Abbott 99c6a82b7f i18n: Clean up logic for picking language in app.
We now have a simple algorithm: First, look at the URL path
(e.g. /de/, which is intended to be an override).  Second, look at the
language the user has specified in their settings.
2018-05-30 08:57:26 -07:00
Shubham Dhama 1d9afc7328 i18n: Pass translation data in page_params.
This is a preliminary step to make translation_data available as a
resource to `i18next.init()`.
2018-05-30 08:57:26 -07:00
Robert Hönig 647c63050f botserver: Add outgoing webhook tokens to botserverrc.
The tokens will be used to authorize the server when sending
messages to the Botserver.
2018-05-30 10:00:19 -04:00
Eeshan Garg 9157b217f0 webhooks/jira: Update docs to conform to style guide.
I spend a lot of time on this. One of our users had reported that
this webhook wasn't working at all. So I tested this with a local
ngrok instance and made sure that it was working. I also took this
opportunity to rewrite the docs for this, which were quite outdated.

With a few changes by Rishi Gupta!
2018-05-29 17:16:52 -07:00
Yashashvi Dave 47aaf4e20a users: Replace duplication with generic func to validate bot id.
This adds a common function `access_bot_by_id` to access bot id within
same realm.  It probably fixes some corner case bugs where we weren't
checking for deactivated bots when regenerating API keys.
2018-05-29 15:47:27 -07:00
Rhea Parekh 1db442acd4 IRC: Add integration documentation. 2018-05-29 10:59:53 -07:00
Rhea Parekh 25e3515d9b Matrix: Add integration documentation. 2018-05-29 10:59:53 -07:00
Shubham Padia 8b8a9be377 upload: Rotate image according to exif data when resizing avatar and emojis.
Fixes the avatar/emoji part of #8177.

Does not address the issue with uploaded images, since we don't do
anything with them.

Also adds 3 images with different orientation exif tags to
test-images.
2018-05-29 10:39:39 -07:00
Tim Abbott 7ccefc3e5d migrations: Remove dependence on PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS.
We don't want to keep around a declaration of
PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS forever, so we should just move
this to a getattr; if the user has set it on their server, we'll use
the value; otherwise, we just use False.
2018-05-29 07:36:53 -07:00
Tim Abbott 91ec0aba09 auth: Improve interactions between LDAPAuthBackend and EmailAuthBackend.
Previously, if you had LDAPAuthBackend enabled, we basically blocked
any other auth backends from working at all, by requiring the user's
login flow include verifying the user's LDAP password.

We still want to enforce that in the case that the account email
matches LDAP_APPEND_DOMAIN, but there's a reasonable corner case:
Having effectively guest users from outside the LDAP domain.

We don't want to allow creating a Zulip-level password for a user
inside the LDAP domain, so we still verify the LDAP password in that
flow, but if the email is allowed to register (due to invite or
whatever) but is outside the LDAP domain for the organization, we
allow it to create an account and set a password.

For the moment, this solution only covers EmailAuthBackend.  It's
likely that just extending the list of other backends we check for in
the new conditional on `email_auth_backend` would be correct, but we
haven't done any testing for those cases, and with auth code paths,
it's better to disallow than allow untested code paths.

Fixes #9422.
2018-05-28 22:47:47 -07:00
Tim Abbott 3842404cc0 ldap: Don't allow password reset for users in LDAP domain.
This is the analog of the last commit, for the password reset flow.
For these users, they should be managing/changing their password in
the LDAP server.

The error message for users doing the wrong thing here is nonexistent
isn't great, but it should be a rare situation.
2018-05-28 22:47:47 -07:00
Tim Abbott 8119670da1 user_settings: Prevent LDAP users from setting a Zulip password.
Previously, if both EmailAuthBackend and LDAPAuthBackend were enabled,
LDAP users could set a password using EmailAuthBackend and continue to
use that password, even if their LDAP account was later deactivated.

That configuration wasn't supported at all before, so this doesn't fix
a pre-existing security issue, but now that we're making that a valid
configuration, we need to cover this case.
2018-05-28 22:47:47 -07:00
Sampriti Panda 250015a5d5 pgroonga: Fix issues with HTML escaping in queries. 2018-05-28 16:53:30 -07:00
Tim Abbott bf9012a559 message: Do future-proofing for public streams with private history.
This should have no effect for now, but it'll make things a bit
simpler in case we make future changes to support public streams
without history public to subscribers (and other organization
members).
2018-05-28 12:25:57 -07:00
Shubham Dhama b410aeb897 message: Limit message access for guest users.
Tweaked by tabbott to interact correctly with the
history_public_to_subscribers feature.
2018-05-28 12:20:32 -07:00
Rhea Parekh 66e3fc4042 import: Clean up the management command.
Significantly tweaked by tabbott because:
* Argparse was already handling the early checks
* Splitting the bottom loop into two loops means we validate all the
input before trying to run actual import code on anything.
* The argparse documentation was confusing about whether the paths
  should be files or directories.
2018-05-28 11:54:04 -07:00
Robert Hönig 497cd4a1f9 outgoing webhooks: Change default URL in all tests.
This reflects the changes to the default URL publicly
displayed to the user. It also changes the default
URL of the default test server outgoing webhook, which
prevented the test server flaskbotrc from working out
of the box.
2018-05-28 10:40:44 -07:00
Yashashvi Dave e82c879b85 custom fields: Add user type of custom fields.
Fixes #8878
2018-05-27 23:01:21 -07:00
Yashashvi Dave 1e948ab405 zerver/lib/users.py: Add function to check given user id is valid.
This function check if given user id exists in realm.
It also check, if user is active and not a bot.
2018-05-27 22:57:55 -07:00
Vishnu Ks c90aac56ba registration: Copy user full_name when importing settings. 2018-05-27 22:39:21 -07:00
Sampriti Panda 018a33ff7c send_test_email: Send email from NOREPLY address as well.
Tweaked by tabbott to also set different subject lines, to make it
easier to see that one got 2 emails.
2018-05-27 22:18:44 -07:00
Rhea Parekh 468afe4840 export: Support export of Custom emojis.
Export of RealmEmoji should also include the image
file of those emojis.

Here, we export emojis both for local and S3 backend
in a method with is similar to attachments and avatars.

Added tests for the same.
2018-05-27 21:54:20 -07:00
Rhea Parekh 7198cc3899 import: Fix RealmEmoji import bug.
RealmEmoji should be imported after UserProfile,
as the new user_profile ids are not allocated
if we import it before.
2018-05-27 21:54:20 -07:00
Rhea Parekh 1b7b9a7164 import: Fix reaction import bug.
In 'zerver_reaction', the emoji_code should be updated
with the RealmEmoji allocated id when the 'reaction_type'
is 'realm_emoji'. Hence we add an extra field 'reaction_field'
in 're_map_foreign_keys', to process the above mentioned
condition.
2018-05-27 21:54:20 -07:00
Rhea Parekh 7a8b853708 Export: Support export of reactions.
We get the reactions from the messages exported.
2018-05-27 21:54:20 -07:00
Rhea Parekh c79d7f1070 Import: Move zerver_reaction from 'messages-000001.json' to 'realm.json'.
Also change the existing slack conversion script structure, to
include 'zerver_realm' in 'realm.json'.
2018-05-27 21:54:20 -07:00
Yago González f84c9b919b api docs: Read parameters and response fixtures from OpenAPI files. 2018-05-26 22:49:55 +02:00
Yago González 30682241c7 api docs: Move OpenAPI files to zerver/openapi. 2018-05-26 21:48:17 +02:00
Tim Abbott 445932cc86 bugdown: Increase limit for number of digits in tweet IDs.
Apprently, they just reached 19 digits.  We set the limit to 30 to
give plenty of headroom for the future.
2018-05-26 10:12:15 -07:00
Yago González b4084d30d2 emails: Move user role logic for followup_day1 to notifications.py. 2018-05-26 07:29:58 -07:00
Yago González d87f0ccb4e api docs: Show Yes/No in the "Required" field of the args table. 2018-05-25 23:31:30 +02:00
Tim Abbott 4e70c9402a export: Fix path logic for exporting avatars with S3 backend.
Apparently, we missed this when we converted the export format to use
longer path names for avatars.
2018-05-25 12:04:34 -07:00
Robert Hönig 7a8c1ec9dc outgoing webhooks: Send additional useful data.
This adds the fields `trigger` and `service_email`
to each message event dispatched by outgoing webhook bots.
`trigger` will be used by the Botserver to determine if
a bot is mentioned in the message.
`service_email` will be used by the Botserver to determine
by which outgoing webhook bot the message should be handled.
2018-05-25 10:33:40 -07:00
Eeshan Garg 0a1b429d04 settings: Remove PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS.
Now that we have a UI for this feature, we don't need this setting
anymore.
2018-05-24 17:18:14 -07:00
Tim Abbott 6967b6519d settings: Add a development-only setting for less dense mode.
This should make it easier for us to iterate on a less-dense Zulip.

We create two classes on body, less_dense_mode and more_dense_mode, so
that it's easy as we refactor to separate the two concepts from things
like colors that are independent.
2018-05-24 12:31:37 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00