Commit Graph

3565 Commits

Author SHA1 Message Date
Tim Abbott 4694de31c5 tests: Fix linter error in test_management_commands. 2019-03-13 14:06:46 -07:00
Hemanth V. Alluri 99c3e2ecdc management: Support sending custom headers when testing a webhook.
this commit adds an option to specify custom headers when using the
`./manage.py send_webhook_fixture_message` tool.
2019-03-13 12:40:47 -07:00
vsvipul e859ab7545 portico: Fix broken electron check condition for password reset.
This logic for passing through whether the user was logged in never
worked, because we were trying to read the client.

Fix this, and add tests to ensure it never breaks again.

Restructured by tabbott to have completely different code with the
same intent.

Fixes #11802.
2019-03-12 11:25:29 -07:00
Harshit Bansal a6e523f9e4 ldap: Ensure email is valid for realm before registering.
Previously, the LDAP authentication model ignored the realm-level
settings for who can join a realm.  This was sort of reasonable at the
time, because the original LDAP auth was an SSO solution that didn't
allow multiple realms, and so one could fully configure authentication
settings on the LDAP side.  But now that we allow multiple realms with
the LDAP backend, one could easily imagine wanting different
restrictions on them, and so it makes sense to add this enforcement.
2019-03-12 11:09:18 -07:00
Challa Venkata Raghava Reddy b69aec2dbc streams: Add first_message_id tracking first message in stream.
This field is primarily intended to support avoiding displaying the
"more topics" feature in new organizations and streams, where we might
know that all messages in the stream are already available in the
browser.

Based on original work by Roman Godov, and significantly modified by
tabbott.

The second migration involved here could be expensive on Zulip Cloud,
but is unlikely to be an issue on other servers.
2019-03-11 13:30:49 -07:00
Rohitt Vashishtha 51986c27a5 markdown: Use common testcases for realmfilters in topics and messages.
The actual bug in #11791 was caused by code reverted in
3ed85f4cd7, so technically #11791 is
already fixed.  However, it makes sense to add tests to ensure that it
doesn't regress in the future as part of closing out the issue.

Fixes #11791.
2019-03-11 11:48:47 -07:00
Tim Abbott c05fb01cbf stream: Fix validator for stream colors.
Apparently, our new validator for stream color having a valid format
incorrectly handled colors that had duplicate characters in them.

(This is caused in part by the spectrum.js logic automatically
converting #ffff00 to #ff0, which our validator rejected).  Given that
we had old stream colors in the #ff0 format in our database anyway for
legacy, there's no benefit to banning these colors.

In the future, we could imagine standardizing the format, but doing so
will require also changing the frontend to submit colors only in the
6-character format.

Fixes an issue reported in
https://github.com/zulip/zulip/issues/11845#issuecomment-471417073
2019-03-11 11:43:39 -07:00
Puneeth Chaganti cb5e9107f4 digest: Directly fetch recipient ids from the DB.
Instead of iterating over Subscriptions and creating the list of home view
recipients, the query now only fetches recipient IDs from the DB.
2019-03-09 23:25:26 -08:00
Harshit Bansal 94649f58f2 tests: Refactor `query_ldap()` and add complete test coverage. 2019-03-09 22:12:51 -08:00
Mateusz Mandera edcb6d57fc email_mirror: Don't remove quotations from forwarded messages.
Addresses point 2 of #10612. We use a regex to detect if a form
of FWD indicator is present at the beginning of the subject, which
means the message has been forwarded.
remove_quotations argument is added to a couple of functions where
it's necessary.
In filter_footer, the criteria for a line to be a possible beginning
of a footer is changed to line.strip() == "--", instead of
line.strip().startswith("--"), because the former would remove
quotations from plaintext emails. This change makes sense, because
RFC 3676 specifies ""-- " as the separator line between the body
and the signature of a message":
https://tools.ietf.org/html/rfc3676
2019-03-09 15:36:17 -08:00
Mateusz Mandera 0633f268fb email_mirror: Move subject processing into process_stream_message.
We remove the 'subject' argument of process_stream_message and make
subject processing happen inside the function, as it's a more
appropriate place than the general process_message function and is
needed to have a good way of disabling removing quotations in forwarded
emails sent into the mirror.
2019-03-09 15:36:17 -08:00
Mateusz Mandera 518ccec235 test_email_mirror: Improve performance of subject stripping tests.
This used to have a single function test_email_subject_stripping which
would run through a sizeable list of example subjects from subjects.json
fixture, form an email with each subject, send it to the email mirror
and check if the resulting stream message has a correctly stripped
topic. That took too much time, because we run through the entire
process_message and most_recent_message codepaths a lot of times.
We change the way of testing to:
1. Ensure process_message applies subject stripping (only need to run
process_message twice here)
2. Test the strip_from_subject function separately, on all the example
from the subjects.json fixtures. This is very fast.
2019-03-09 23:15:41 +01:00
Mateusz Mandera 0d84be8e4b test_email_mirror: Clean up unnecessary use of "type: Any" on MIMEText.
The explanation on those type declarations referenced issue
https://github.com/python/typeshed/issues/275,
which has now been resolved.
2019-03-09 23:15:16 +01:00
YashRE42 a724a38c03 markdown: Improve handling of broken img urls.
Some urls which end with image file extensions (eg .jpg) may link to
html pages. This adds handling for linx.li, wikipedia.org and
pasteboard.co. If it is possible, we redirect to the actual image url
otherwise we do not attempt to render it as an image.

Fixes #10438.
2019-03-08 13:39:34 -08:00
Vishnu Ks a288cfc43a uploads: Show used upload space in attachments UI. 2019-03-07 20:18:00 -08:00
Mateusz Mandera dbff533e09 email_mirror: Add the sender at the start of stream message.
Fixes part 3 of #10612. When sending an email to the email mirror to a
stream address, if "+show-sender" is added in the address, the stream
message will now include "From: <sender>" at the top.
2019-03-07 14:28:33 -08:00
Pragati Agrawal 60c7467464 test_events: Add test for `do_change_logo_source` event.
This adds tests for `do_change_logo_source` event in both day as well as
night mode in `test_event.py`.
2019-03-07 12:32:30 -08:00
Tim Abbott 99a2c21ff3 test_events: Fix incorrect usage of get_realm.
The test_events system was in several tests using get_realm to fetch a
realm object, rather than accessing self.user_profile.realm.  This
created subtle problems where we were neither directly editing nor
refreshing the `realm` object associated with our UserProfile object
from the database after our the `do_*` methods.

The payoff for this is we can update the previously confused
`do_change_icon_source` test to actually change the state and have the
correct result.
2019-03-07 12:29:16 -08:00
Pragati Agrawal 095e24c515 realm_logo: Update `upload_logo` views to return nothing. 2019-03-07 12:12:00 -08:00
Pragati Agrawal 276020445d realm_logo: Make `delete_logo_backend` views return nothing. 2019-03-07 12:12:00 -08:00
Pragati Agrawal 78e9ad3128 test_upload: Use actions.py do-change functions to change property. 2019-03-07 12:12:00 -08:00
Pragati Agrawal 0192f80d2e org_settings: Remove print statements.
This commit deletes the redundant `print` statement from the
`test_upload.py` file.
2019-03-07 12:12:00 -08:00
Rohitt Vashishtha 3ed85f4cd7 Revert "bugdown: Process word boundaries properly in realm_filters."
This reverts commit ff90c0101c but keeps
the test cases added for reference.

This was reverted because it was both not a clean solution and created
other realm filters bugs involving dashes (etc.).
2019-03-07 11:03:35 -08:00
Harshit Bansal 4a9bd89f47 ldap: Continue syncing other fields even if a field is missing.
Earlier the behavior was to raise an exception thereby stopping the
whole sync. Now we log an error message and skip the field. Also
fixes the `query_ldap` command to report missing fields without
error.

Fixes: #11780.
2019-03-05 16:19:27 -08:00
overide 58d28eed5d markdown: Fix emojis not rendering with :bogus: in the line.
This fixes an issue where invalid emoji name prevents following
emojis from rendering.

This reverts the code change in
8842349629, while still passing the
tests added in that commit (it seems the original commit had
misdiagnosed an ordering bug and thus introduced this issue).

Fixes: #11770.
2019-03-05 16:05:25 -08:00
Vishnu Ks 8356804bf1 api: Add endpoint for fetching used upload space info. 2019-03-04 18:46:13 -08:00
Vishnu Ks ef3fbfeb98 events: Include realm_plan_type in initial state data. 2019-03-04 18:46:13 -08:00
Vishnu Ks 4eb5b9ad80 realm: Restrict upload quota according to plan. 2019-03-04 18:46:13 -08:00
Vishnu Ks 94ae2dc24e models: Cache currently_used_upload_space_bytes function. 2019-03-04 18:46:13 -08:00
Vishnu Ks cc9f00f53e events: Include realm_upload_quota in initial state data. 2019-03-04 18:46:13 -08:00
Vishnu Ks e522308507 models: Move currently_used_upload_space function to Realm model. 2019-03-04 18:46:13 -08:00
Tim Abbott 73655a6176 test_auth_backends: Disable Internet for httpretty.
This makes debugging issues when using httpretty a lot more
convenient.
2019-03-04 12:13:49 -08:00
Tim Abbott 6b796a1e8c test_auth_backends: Refactor some GitHub tests to be generic social.
This is preparation to being able to run these tests automatically
for the Google auth backend as well.
2019-03-04 12:13:11 -08:00
Harshit Bansal 216b7b0a19 auth: Remove `invalid_subdomain` restriction from LDAP backend.
Fixes: #11692.
2019-03-04 11:53:53 -08:00
Tim Abbott 16123c9a58 realm_logo: Fix synchronization of realm night logo.
The night logo synchronization on the settings page was perfect, but
the actual display logic had a few problems:

* We were including the realm_logo in context_processors, even though
  it is only used in home.py.
* We used different variable names for the templating in navbar.html
  than anywhere else the codebase.

* The behavior that the night logo would default to the day logo if
  only one was uploaded was not correctly implemented for the navbar
  position, either in the synchronization for updates code or the
  logic in the navbar.html templates.
2019-03-02 09:45:50 -08:00
Bennet Sunder 7c5f316cb8 alert_words: Performance improvements in looking for alert_words.
This commit leverages the ahocorasick algorithm to build a set of user_ids
that have their alert_words present in the message. It runs in linear time
of the order of length of the input message as opposed to number of
alert_words. This is after building a ahocorasick Automaton which runs
in O(number of alert_words in entire realm) which is usually cached.
2019-03-01 15:36:39 -08:00
overide 0dcfc22406 markdown: Fix numbered list handling of blank lines between blocks.
This fixes an issue where blank lines between blocks were causing
auto-numbering of list to stop before the blank line resulting
in two separate numbered list instead of one.

Edited significantly by tabbott to explain the tricky details in the
comments.

Fixes: #11651.
2019-03-01 15:29:07 -08:00
Rafid Aslam 9038230de8 decorator.py: Add `max_int_size` param to `to_non_negative_int()`.
Add `max_int_size` parameter to `to_non_negative_int()` in
decorator.py so it will be able to validate that the integer doesn't
exceed the integer maximum limit.

Fixes #11451
2019-03-01 11:16:46 -08:00
Tim Abbott 9693d8a651 docs: Extend testing discussion in events system doc.
Importantly, this also makes that section linkable.
2019-03-01 09:23:56 -08:00
Eeshan Garg 9252c43225 api/streams: Support including bot owner's subscriptions.
This is important for situations such as with our Zapier app,
where the requesting user may be a bot that would like to access
its owner's subscriptions.

Tweaked by tabbott to eliminate the 2^N growth of cases in
do_get_streams.
2019-02-28 22:32:05 -08:00
Raymond Akornor 4603cdba7e tests: Optimize a slow test in tests_bots.py.
tests now ran in 7.649s from 9.297s.  And this test works just as well
with 3 bots, since only 3 database queries with 3 bots confirms we're
not doing linear queries in the number of bots in the organization.
2019-02-28 22:01:55 -08:00
Tim Abbott de65a04ae0 streams: Disable inline URL preview when rendering stream descriptions.
We want to use the baseline features of bugdown, but not fancy things
like inline URL previews, since the whole structure of stream
descriptions is to have a single-line thing supporting some
formatting.

The migration part of this change fixes a bug encountered by some
organizations upgrading from older versions of Zulip.
2019-02-28 17:00:40 -08:00
Tim Abbott d6c09eac51 bugdown: Add support for no_previews argument.
This allows us to have some features using bugdown rendering where
inline image previews will not be rendered (which would be problematic
for e.g. stream descriptions).
2019-02-28 16:54:04 -08:00
Tim Abbott 62dc6dda49 guests: Block guest user access to default streams.
Guest users will just get an empty list of default streams; we also
hide the "Default streams" organization view from the guest users UI.

This is for consistency with not providing guest users the full list
of streams in an organization.
2019-02-28 16:36:30 -08:00
Tim Abbott 1ce0e8256b zoom: Avoid sending Zoom API secret to other admin clients.
Fixing this involves fixing the backend to handle unchanged field
submissions of the Zoom credentials without trying to re-validate the
credentials (for performance) as well as to fetch the already-sent
secret.
2019-02-28 15:43:42 -08:00
Rishi Gupta d833c70dc7 org settings: Explain Zoom support is experimental, and fix a few strings.
Visually, #zoom_help_text acts like
.organization-settings-parent div:first-of-type when the Zoom option
is selected, but isn't treated as such.

No visual change with the #google_hangouts_domain change; just there to make
the code more readable/defensible.
2019-02-28 15:09:35 -08:00
Greg Price 9869153ae8 push notif: Send a batch of message IDs in one `remove` payload.
When a bunch of messages with active notifications are all read at
once -- e.g. by the user choosing to mark all messages, or all in a
stream, as read, or just scrolling quickly through a PM conversation
-- there can be a large batch of this information to convey.  Doing it
in a single GCM/FCM message is better for server congestion, and for
the device's battery.

The corresponding client-side logic is in zulip/zulip-mobile#3343 .

Existing clients today only understand one message ID at a time; so
accommodate them by sending individual GCM/FCM messages up to an
arbitrary threshold, with the rest only as a batch.

Also add an explicit test for this logic.  The existing tests
that happen to cause this function to run don't exercise the
last condition, so without a new test `--coverage` complains.
2019-02-26 16:41:54 -08:00
Greg Price 28ff9670de push notif: Push `gcm_options` logic inside "payload" helpers.
These are logically closely related.
2019-02-26 16:41:54 -08:00
Greg Price 8f26e12c85 push notif: Clarify get_*_payload, and factor another out.
This is a pure refactor; adding docstrings, making some names more
explicit, and pulling out one small helper.
2019-02-26 16:41:54 -08:00
Greg Price 69ded8b1b4 push notif: Drop irrelevant fields in `remove` payloads.
These fields don't make much sense in this case; and the client
doesn't look at them and never has.  Stop including them.
2019-02-26 16:41:54 -08:00