Commit Graph

12003 Commits

Author SHA1 Message Date
Rohitt Vashishtha fb2946aaf6 Revert "markdown: Remove paragraphs that only contain a tweet link."
This reverts commit d3770153a6.

We do not show a link to the tweet in our preview, so we should revert
to our previous behavior for now.
2020-07-17 14:30:22 -07:00
Tim Abbott 6be3fca037 auth: Remove short_name from LDAP API.
As best I can tell, we fetched this field and then ignored it, so
unlike the last few commits, this is more a code cleanup than a
functional change.
2020-07-17 11:28:08 -07:00
Steve Howell c44500175d database: Remove short_name from UserProfile.
A few major themes here:

    - We remove short_name from UserProfile
      and add the appropriate migration.

    - We remove short_name from various
      cache-related lists of fields.

    - We allow import tools to continue to
      write short_name to their export files,
      and then we simply ignore the field
      at import time.

    - We change functions like do_create_user,
      create_user_profile, etc.

    - We keep short_name in the /json/bots
      API.  (It actually gets turned into
      an email.)

    - We don't modify our LDAP code much
      here.
2020-07-17 11:15:15 -07:00
Steve Howell c60f4236a9 api: Do not require short_name to create user.
When you post to /json/users, we no longer
require or look at the short_name parameter,
since we don't use it in any meaningful way.

An upcoming commit will eliminate it from the
database.
2020-07-17 11:15:15 -07:00
Steve Howell b375581f58 api: Remove (sender_)short_name from message APIs. 2020-07-17 11:15:14 -07:00
Steve Howell e6974d3013 reaction tests: Clean up optional parameters.
This fixes up some complex helpers that may
have had some value before f-strings come along,
but they mostly obscured the logic for
people reading the tests.

We still keep really simple helpers for the
common cases, but there are no optional
parameters for them.

One goal of this fix is to remove the
short_name concept, and we just explicitly
set senders everywhere we need them.

We also now have each test just explicitly set
its reaction_type.

For cases where we have custom message ids
or senders, we just inline the simple call
to api_post.
2020-07-17 11:04:54 -07:00
Steve Howell 3b2c881ce6 tests: Decouple test_retention and test_reactions.
We generally want to avoid having two sibling test
suites depend on each other, unless there's a real
compelling reason to share code.  (And if there is
code to share, we can usually promote it to either
test_helpers or ZulipTestCase, as I did here.)

This commit is also prep for the next commit, where
I try to simplify all of the helpers in EmojiReactionBase.

Especially now that we have f-strings, it is usually
better to just call api_post explicitly than to
obscure the mechanism with thin wrappers around
api_post.  Our url schemes are pretty stable, so it's
unlikely that the helpers are actually gonna prevent
future busywork.
2020-07-17 11:04:54 -07:00
Steve Howell f13327223d tests: Extract get_user_id helper for import tests.
The new helper sets us up to drop short_name.
2020-07-17 11:04:54 -07:00
Steve Howell 0a88202763 webhooks: Ignore short_name to guess user.
The short_name field isn't super reliable, and
we are about to eliminate it.
2020-07-17 11:04:54 -07:00
Steve Howell 7d9f24c58c mypy: Fix Realm types in test_import_export.
It's not clear to me why these passed mypy
before, given this:

    def assert_realm_values(f: Callable[[Realm], Any], ...

But this is clearly more accurate.
2020-07-17 11:04:54 -07:00
Steve Howell 2a9cb4c8cf dead code: Remove legacy reaction actions. 2020-07-17 04:40:01 -04:00
Steve Howell 2374e25b94 import: Import AlertWord table. 2020-07-16 08:50:31 -07:00
Steve Howell 318c55e030 export: Export AlertWord table. 2020-07-16 08:50:31 -07:00
Anders Kaseorg c2f9db4602 logo: Update Zulip logo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-16 01:37:08 -07:00
Tim Abbott 289819fb70 push_notifications: Reduce log level for weird warning.
This issue isn't something a system administrator needs to take action
on -- it's a likely minor logic bug around organization
administrators moving topics between streams.

As a result, it shouldn't send error emails to administrators.
2020-07-16 01:22:03 -07:00
Rohitt Vashishtha b64ba98e90 markdown: Use unicode ellipses for collapsing spoilers.
We had initially implemented this feature using `(...)` but `(…)` is the
better variation.
2020-07-15 23:30:28 -07:00
Rohitt Vashishtha 7d5d90b484 markdown: Format spoilers for emails.
This is a hacky fix to avoid spoiler content leaking in emails. The
general idea here is to tell people to open Zulip to view the actual
message in full.

We create a mini-markdown parser here that strips away the fence content
that has the 'spoiler' tag for the text emails.

Our handling of html emails is much better in comparison where we can
use lxml to parse the spoiler blocks.
2020-07-15 23:30:28 -07:00
Tim Abbott 2f66c825a2 push_notifications: Disable badge counts.
We include tests for the new implementation to avoid churning the
codebase too much so this can be easily reverted when we are able to
re-enable the feature.
2020-07-15 22:19:53 -07:00
Steve Howell 7c0fa3aefc populate_db: Add alert words for Zulip users. 2020-07-15 12:33:08 -07:00
Steve Howell 63b15b77a3 alert_word tests: Use hamlet for all tests.
The tests had a bunch of different ways to create
users; now we are consistent.  (This is a bit of
a prep step, too, to allow us to easily clean
Hamlet's existing words before each test.)
2020-07-15 12:33:08 -07:00
Rohitt Vashishtha 544c1cd3fe test-api: Use dynamic message ids in JS examples.
We replace hardcoded message ids which was prone to failure. This
new approach makes sure that the messages we're using for tests
actually exist.
2020-07-15 12:33:08 -07:00
Rohitt Vashishtha 1a9a478e5d markdown: Assert we handle timestamps sensibly in push notifications.
We could certainly do better with the handling here, but using the raw
string that the user gave us is okayish for now.

Proper formatting of timestamps requires handling locales and timezones
of the receiver as well which is a larger project.
2020-07-15 11:18:32 -07:00
Rohitt Vashishtha 78c48935ca markdown: Format spoilers for push notifications.
We now do something sensible for spoilers in notifications. A message
like:

    ```spoiler Luke's father is
    Vader. Don't tell anyone else.
    ```

would be rendered as:

    Luke's father is (...)
2020-07-15 11:17:38 -07:00
Tim Abbott 7ade8fae78 test_message_send: Fix invalid UserProfile creation. 2020-07-15 10:19:41 -07:00
Mateusz Mandera b9b146c809 migrations: Add case-insensitive unique indexes on realm and email.
Tweaked by tabbott to use clearer conventions and to document it in
the upgrade notes.

Fixes #15772.
2020-07-15 09:42:40 -07:00
Anders Kaseorg 2794bc1ef4 lint: Reformat YAML files with Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-14 16:25:31 -07:00
orientor 40f76bb030 openapi_py: Add function for validating requests.
Add a function for validating requests made during tests in `openapi.py`.
2020-07-14 14:23:58 -07:00
orientor bff8e43729 openapi_py: Add openapi-core validator object to OpenAPISpec class.
OpenAPISpec is our main class for accessing OpenAPI objects and
has the capability of creating the OpenAPI objects only once, thus
saving time. Since the openapi-core request validator object is going
to be accessed for considerable time during testing, hence add it to
the class for faster testing.
2020-07-14 13:57:50 -07:00
orientor ade5dae564 openapi: Fix `/settings/notifications` OpenAPI documentation.
We pass these values encoded as JSON and parsed with
validator=check_int, not as raw integers.
2020-07-14 13:57:50 -07:00
orientor ee39522eef openapi: Improve `/messages/{message_id}` PATCH documentation.
We extract OptionalContent and RequiredContent since some endpoints
require it and other don't.

In an ideal world, we'd have a better way to express these small
variants.
2020-07-14 13:57:50 -07:00
orientor 4ca4d43b0c openapi: Improve `/message` GET OpenAPI spec.
The `num_after` and `num_before` parameters should be 0 or greater.
2020-07-14 13:57:50 -07:00
orientor bb405ebc3d openapi: Move `/users/{user_id}` below endpoints with similar regex.
openapi-core, the request validator has a bug due to which data type
of path parameters is not checked. Hence `/users/{user_id}` can match
with `users/me`. So change the position of`/user/{user_id}` after all
such possible matches to avoid errors.

See https://github.com/p1c2u/openapi-core/issues/226 for details.
2020-07-14 13:57:50 -07:00
orientor 26bcd7e4d1 openapi: Move `/messages/{message_id}` below endpoints with similar regex.
openapi-core, the request validator has a bug due to which data type
of path parameters is not checked. Hence `/messages/{message_id}`
can match with `messages/matches_narrow`. So change the position of
of `message/{message_id}` after all such possible matches to avoid
errors.

See https://github.com/p1c2u/openapi-core/issues/226 for details.
2020-07-14 13:57:50 -07:00
Aman Agrawal e22885a6bf push_notifications: Return if push_notify already active.
If the push_notification for the UserMessage is already active,
we don't send any push notification to the user. This may
happen due to race conditions.

Added and fixed test cases affected by this.
2020-07-14 00:35:29 -07:00
Anders Kaseorg 6189e4d0c1 python: Convert more percent formatting to "".format.
Semgrep has gotten a little more clever at applying the percent
formatting rule.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-13 13:16:38 -07:00
Mateusz Mandera c231d88d9f upgrade: Add management command to fix FTS indexes.
Upgrading the base OS's dictionary files can corrupt our FTS
indexes. We add a command for fixing them.

Fixes #14982.
2020-07-13 12:40:44 -07:00
Rohitt Vashishtha 37c0577f8a markdown: Assert twitter previews render inside spoilers.
Previously, we rendered the twitter previews outside of a
spoiler block at the end of the message. The commit series
ending with this commit fixes that by inlining twitter
previews instead of appending them all at the end. As a
consequence of the inlining, we have fixed the issue here.
This commit just adds a test to assert that.

Fixes #15518.
2020-07-13 12:24:32 -07:00
Rohitt Vashishtha d3770153a6 markdown: Remove paragraphs that only contain a tweet link.
This is similar to our behavior with image previews, and helps
reduce clutter in the final rendered html.

We add the string 'Tweet: ' to our existing tests so those tests
remain the same.
2020-07-13 12:24:32 -07:00
Rohitt Vashishtha 87e01cd1fa markdown: Inline Twitter previews instead of appending at end.
This commit makes our handling of twitter previews consistent with
how we handle our inline images so that tweets render next to the
paragraph that links to the tweet.
2020-07-13 12:24:32 -07:00
Rohitt Vashishtha a8ab745ee4 markdown: Extract get_inlining_information for link previews.
We decouple the logic of insertion rules for inline links from
image preview logic. Now, we can use this same logic for other
kinds of link previews as well.
2020-07-13 12:24:32 -07:00
Steve Howell 9662af842f export: Remove stream sanity check.
We also remove the post_process_data option.

The sanity check is just overkill at this point,
since the mechanism to find streams is very
direct due to a recent commit.
2020-07-09 11:34:00 -07:00
Steve Howell 54c596cfc4 export: Just export all streams in a realm.
Before this change we would only export streams
that had actual subscribers, which is usually
harmless, but it was mostly a relic of a one
time migration that we did when we were cleaning
up some dirty data in some of our very early
databases (circa 2016).

Now we work down the table hierarchy in a
more natural way:

    - get Streams in Realm
    - get Recipients matching above Streams
    - get Subscriptions matching above Recipients

Note that for per-user exports, I kept the
same logic (users -> subscriptions -> recipients ->
streams) we had before.

One subtle detail here is that we make our
final Config blocks--which build the final
version of Recipient/Subscription--now hang
off of realm_config.

Fixes #15146.
2020-07-09 11:33:55 -07:00
Mohit Gupta 79243f7d52 tests: Move test_near_pm_message_url to test_message_fetch.py. 2020-07-08 14:00:05 -07:00
Mohit Gupta 16c3121bc8 tests: Move test_is_private_flag_not_leaked to test_message_flags.py. 2020-07-08 14:00:05 -07:00
Mohit Gupta b0b3cf86ce tests: Move test_auto_subbed_to_personals test to test_signup.py.
This is really a test about account creation.
2020-07-08 13:59:56 -07:00
Mohit Gupta 478956ff0b tests: Move MiscMessageTest to test_audit_log.py. 2020-07-08 13:53:56 -07:00
Mohit Gupta f03c5e35e4 tests: Move NoRecipientIDsTest to test_subs.py. 2020-07-08 13:51:32 -07:00
Mohit Gupta 086e9e281b models: Remove to_log_dict method of Message model as it's unused. 2020-07-07 17:14:10 -07:00
Mohit Gupta de81b92dee tests: Move MessageVisibilityTest to test_message_fetch.py. 2020-07-07 17:14:10 -07:00
Mohit Gupta 012622aa4a tests: Move CheckMessageTest to test_message_send.py. 2020-07-07 17:14:10 -07:00