Commit Graph

9256 Commits

Author SHA1 Message Date
Pragati Agrawal e1772b3b8f tools: Upgrade Pycodestyle and fix new linter errors.
Here, we are upgrading pycodestyle version from 2.4.0 to 2.5.0.

Fixes: #11396.
2019-01-31 12:21:41 -08:00
Vishnu Ks 0d0007742f requirements: Upgrade pika from 0.12.0 to 0.13.0.
The important changes to pika for us are based on this PR of ours:
https://github.com/pika/pika/pull/1129

Fixes #11394.
2019-01-31 10:04:07 -08:00
Harshit Bansal 837483b026 tests: Don't fail silently on 500 on logins. 2019-01-29 16:01:30 -08:00
Harshit Bansal 6777b94d41 lib: Extract `validate_user_custom_profile_field()`. 2019-01-29 16:01:30 -08:00
Harshit Bansal 9bac192241 dev_ldap: Add custom profile data. 2019-01-29 16:01:30 -08:00
Harshit Bansal 88c1d868d2 dev_ldap: Fix password in mode 'c' to be equal to username. 2019-01-29 16:01:30 -08:00
Harshit Bansal dccfadc1a5 minor: Fix some typing errors. 2019-01-29 16:01:30 -08:00
Rishi Gupta 78d98168a4 portico: Add stride import to /features and /help. 2019-01-29 13:48:39 -08:00
Steve Howell c2fcfc087a bugdown: Include message id in exceptions. 2019-01-29 12:49:56 -08:00
Steve Howell 643d4cacd3 bugdown tests: Pass message stub to bugdown_convert().
This makes the calling sequence slightly more
realistic (without any additional overhead).
2019-01-29 12:49:56 -08:00
Tim Abbott 8d404dbcc6 confirmation: Fix "continue to registration" full name handling.
Apparently, the "continue to registration" flow used a subtly invalid
way of encoding the full name.  We put in the query part of the action
URL of the HTML form, but apparently HTML forms with a `GET` type will
ignore the query part (replacing it with any input values), which
makes sense but doesn't do what we want here.  There are a few sane
ways to fix it, but given that the encoding logic we had before for
including the name in the URL was ugly, I'm pretty happy with just
adding a hidden input to the form for the name.
2019-01-29 12:39:40 -08:00
Tim Abbott 9dc4b04e52 auth: Migrate Google authentication off deprecated name API.
As part of Google+ being removed, they've eliminated support for the
/plus/v1/people/me endpoint.  Replace it with the very similar
/oauth2/v3/userinfo endpoint.
2019-01-29 12:39:40 -08:00
Tim Abbott aea93f9826 api: Document support for setting stream description on creation.
This makes clear how to set the stream description when making a new
stream.

Fixes #11372.
2019-01-29 11:28:55 -08:00
Daniel Bradburn 1e65cdd072 emoji: Only resize custom emoji that need it.
This additional logic to prevent resizing is certain circumstances
(file size, dimensions) is necessary because the pillow gif handling
code seems to be rather flaky with regards to handling gif color
palletes, causing broken gifs after resizing.  The workaround is to
only resize when absolutely necessary (e.g. because the file is larger
than 128x128 or 128KB).

Fixes #10351.
2019-01-29 10:33:50 -08:00
Rohitt Vashishtha a2751d90c2 widgets: Remove ALLOW_SUB_MESSAGES flag and enable widgets by default. 2019-01-29 09:34:14 -08:00
Rohitt Vashishtha c176891c2e poll-widget: Refactor comment to option.
We had initially designed the poll widget like a blog
post with comments beneath it but it makes more sense
to think of it as just a simple poll with options.
2019-01-29 09:34:14 -08:00
Rohitt Vashishtha 5641afc6e2 poll-widget: Add syntax for adding options when creating poll.
We add a new syntax which converts the messages like the following:

```
/poll Who do you support?

Nadal
- Djokovic
```

to a poll with the two names as options. The list syntax is optional
since anyone making a poll is likely to want to create a list anyway.
2019-01-29 09:34:14 -08:00
Tim Abbott 0181086d0f streams: Improve validation for arguments to stream creation.
This doesn't have any security impact, since we overwrote any other
fields in any case, and also this step happens before the security
part of input validation for stream creation.  But this does improve
error messages if one tries to specify other arguments, and also makes
more clear that the `description` argument is supported here.
2019-01-28 22:02:16 -08:00
Akshay Gupta 88ed75d06d travis_webhooks: Update the list of bad statuses.
Fixes #11259.
2019-01-28 21:37:18 -08:00
Adarsh Patel 9b740df277 api_docs: Specify max width for long code examples.
Fixes #10673.
2019-01-28 22:39:54 -03:30
Rishi Gupta 7c11fe819a billing: Update license ledger when users are added and removed. 2019-01-28 16:04:11 -08:00
Wyatt Hoodes 8eac361fb5 docs: Refactor BS work with use of cache_with_key.
Refactor the potentially expensive work done by Beautiful Soup into a
function that is called by the alter_content function, so that we can
cache the result.  Saves a significant portion of the runtime of
loading of all of our /help/ and /api/ documentation pages (e.g. 12ms
for /api).

Fixes #11088.

Tweaked by tabbott to use the URL path as the cache key, clean up
argument structure, and use a clearer name for the function.
2019-01-28 15:21:52 -08:00
Matthew Wegner 370cf1a2cb import: Normalize Slackbot String Comparison.
In very old Slack workspaces, slackbot can appear as "Slackbot", and
the import script only checks for "slackbot" (case sensitive).  This
breaks the import--it throws the assert that immediately follows the
test.  I don't know how common this is, but it definitely affected our
import.

The simple fix is to compare against a lowercased-version of the
user's full name.
2019-01-28 14:59:41 -08:00
Steve Howell f2b26b9bcc integration tests: Add check for new messages.
This commit should make it a bit easier to debug
integrations that silently don't send any messages.
2019-01-28 14:56:42 -08:00
Rohitt Vashishtha ff90c0101c bugdown: Process word boundaries properly in realm_filters.
Earlier, our realm filters didn't render for languages that do not
use spaces (eg: Japanese) since we used to check for the presence
of an actual space character. This commit replaces that logic with
a complex scheme to detect word boundaries.

Also, we convert the RealmFilterPattern to subclass InlineProcessor
and make use of the new no-op feature in py-markdown 3.0.1 where we
can tell py-markdown that our pattern didn't find a match despite
the initial regex getting matched.

Fixes #9883.
2019-01-28 14:48:15 -08:00
Steve Howell ad071ced47 bugdown: Avoid recomputing the stream-link regex. 2019-01-28 13:12:37 -08:00
Rohitt Vashishtha 2dc447d707 bugdown: List py-markdown 3.0.1 features that we do not use.
Tweaked by tabbott to extend the documentation.
2019-01-28 13:12:37 -08:00
Rohitt Vashishtha 434094e599 bugdown: Restructure Bugdown to extend Markdown from being an extension.
Since we are building our parser from scratch now:

1. We have control over which proccessor goes at what priority number.
   Thus, we have also shifted the deprecated `.add()` calls to use the
   new `.register()` calls with explicit priorities, but maintaining
   the original order that the old method generated.

2. We do not have to remove the processors added by py-markdown that
   we do not use in Zulip; we explicitly add only the processors we
   do require.

3. We can cluster the building of each type of parser in one place,
   and in the order they need to be so that when we register them,
   there is no need to sort the list. This also makes for a huge
   improvement in the readability of the code, as all the components
   of each type are registered in the same function.

These are significant performance improvements, because we save on
calls to `str.startswith` in `.add()`, all the resources taken to
generate the default to-be-removed processors and the time taken to
sort the list of processors.

Following are the profiling results for the changes made. Here, we
build 10 engines one after the other and note the time taken to build
each of them. 1st pass represents the state after this commit and 2nd
pass represent the state after some regex modifications in the commits
that follow by Steve Howell. All times are in microseconds.

| nth Engine | Old Time | 1st Pass | 2nd Pass |
| ---------- | -------- | -------- | -------- |
|          1 |  92117.0 |  81775.0 |  76710.0 |
|          2 |   1254.0 |    558.0 |    341.0 |
|          3 |   1170.0 |    472.0 |    305.0 |
|          4 |   1155.0 |    519.0 |    301.0 |
|          5 |   1170.0 |    546.0 |    326.0 |
|          6 |   1271.0 |    609.0 |    416.0 |
|          7 |   1125.0 |    459.0 |    299.0 |
|          8 |   1146.0 |    476.0 |    390.0 |
|          9 |   1274.0 |    446.0 |    301.0 |
|         10 |   1135.0 |    451.0 |    297.0 |
2019-01-28 13:12:37 -08:00
Rohitt Vashishtha 9f2c52c86e bugdown: Rename variables regex to REGEX for importing regex module. 2019-01-28 12:00:58 -08:00
Steve Howell 3b7d899532 bugdown: Use CompiledPattern in AtomicLinkPattern.
We avoid re-computing the regex string here, and we
also avoid re-compiling the regex itself.

I decided to put the "one_time" decorator in the
bugdown file itself, just to reduce friction in
folks reading the "buyer beware" comments.

Unfortunately, we can't use this for the
get_web_link_regex() function due to testing concerns,
so that continues to do an inelegant cache-with-global-var
scheme.
2019-01-28 11:58:47 -08:00
Steve Howell eea711a805 bugdown: Flatten get_web_link_regex().
We use early-exit to flatten the code.

I also tweaked the comments a bit based on some recent
profile findings.  (e.g. reading the file isn't actually
a big bottleneck, it's more the regex itself)
2019-01-28 11:58:46 -08:00
Steve Howell 852756aeb3 bugdown: Eliminate LinkPattern class.
The only code that used LinkPattern was AtomicLinkPattern.

We just move the helper method into AtomicLinkPattern.
2019-01-28 11:58:05 -08:00
Steve Howell 77446a710c bugdown: Optimize CompiledPattern.
We don't need the hacky step of passing in a blank
regex to the superclass's __init__ function.

All we need to do is assign `md` to `self.md`.
2019-01-28 11:57:28 -08:00
Steve Howell 025df33d7a bugdown: Rename VerbosePattern to CompiledPattern.
This class only requires a compiled regex--it's up
to the callers how they compile it (verbose or
otherwise).
2019-01-28 11:56:01 -08:00
Eeshan Garg 681f07f98f check_message: Refactor stream validation into separate function. 2019-01-28 11:48:17 -08:00
Eeshan Garg 4cb9a472e6 addressee: Add helper to check for missing or empty topic. 2019-01-28 11:42:26 -08:00
kunal-mohta b89769420a stream_color: Extend subscribe API to support color attribute.
This fixes an annoying bug where clicking to subscribe to a stream
would change the color shown in the "manage streams" UI immediately
after you click.

Fixes #11072.
2019-01-25 16:44:59 -08:00
Pragati Agrawal 1e811b42ec org settings: Add realm level setting for missed message content in email.
This adds a setting under "Notification" section of
"Organization settings" tab, which enables Organization administrator to
control whether the missed message emails include the message content or
not.

Fixes: #11123.
2019-01-25 14:34:10 -08:00
Vishnu Ks bec875a9af import realm: Use processes for resizing avatar images.
This should significantly improve the data import performance when
importing large open source realms from Slack.

Fixes #11009.
2019-01-25 12:37:12 -08:00
Mohit Gupta c89356fabf messages: Fix 500 error with duplicate delete message requests.
Multiple delete message requests for the same message sometimes caused
a 500 error.  This happened via the normal IntegrityError being thrown
by delete message/archiving code.

This was manually reproduced by adding latency in function
move_messages_to_archive() in retention.py and
delete_message_backend() in views.py.  This addresses the problem by
adding code to handle the exception and throw JsonableError to convert
500 to 400 errors, with an automated test.
2019-01-25 11:07:02 -08:00
Tim Abbott a64c434b9e topics: Fix messages_for_topic to use iexact matching.
This fixes the archive codebase to use our covention of using
case-insensitive matching for message topics.
2019-01-24 17:55:15 -08:00
Eeshan Garg 5057406f16 webhooks/clubhouse: Extract story labels from actions.
Apparently, Clubhouse has two different payloads for story label
changes, one where the label name lives inside the "references"
object, and the other where it lives inside the "actions" object.
Their webhook API is still in beta, so this could just be a bug.
Anyhow, we should support both.
2019-01-24 20:08:59 -03:30
Vaibhav 268c25e95f streams: Prevent members from creating admin-only streams.
This a check on server side to verify whether the user sending request
to create stream where only admins can post is an admin or not; Raises
a JsonableError when the user is not the realm admin.
2019-01-24 10:51:45 -08:00
Eeshan Garg 2b81d66537 webhooks/clubhouse: Add support for GitHub handler events.
These events are triggered when one has Clubhouse's GitHub
integration and custom event handlers set up.
2019-01-24 10:01:17 -08:00
Steve Howell 0ef5d1f9c8 user status: Add "status_text" support in the backend.
You can now pass in an info field with a value
like "out to lunch" to the /users/me/status,
and the server will include that in its outbound
events.

The semantics here are that both "away" and
"status_text" have to have defined values in order
to cause changes.  You can omit the keys or
pass in None when values don't change.

The way you clear info is to pass the empty
string.

We also change page_params to have a dictionary
called "user_status" instead of a set of user
ids.  This requires a few small changes on the
frontend.  (We will add "status_text" support in
subsequent commits; the changes here just keep
the "away" feature working correctly.)
2019-01-24 09:46:35 -08:00
Steve Howell ac861f2b7d user status: Restructure code.
We now have single function that handle both away
and not-away.

This refactoring sets us up to piggyback "info" more
easily onto status updates.

The only thing that changes here is that we don't
delete database rows any more when users revoke
their away status.  Instead we just set the status
to NORMAL.
2019-01-24 09:46:35 -08:00
Steve Howell 7e904c3a1a user status: Add status_text field to the database.
The status_text field will be something like "out to lunch."
2019-01-24 09:46:35 -08:00
sameerchoubey d4cd789213 webhooks/stripe: Add support for remaining payout events.
This adds support for payout events to the Stripe
intergration.
2019-01-23 21:29:40 -03:30
kunal-mohta c07f85250d messages: Extend do_delete_message to do_delete_messages.
do_delete_message has been renamed to do_delete_messages and all
occurrences of the function replaced with the appropriate new version.
2019-01-23 16:49:12 -08:00
kunal-mohta 67f2533ba3 messages: Rename do_delete_messages to do_delete_messages_by_sender. 2019-01-23 16:49:12 -08:00