Commit Graph

30248 Commits

Author SHA1 Message Date
Wyatt Hoodes 8eac361fb5 docs: Refactor BS work with use of cache_with_key.
Refactor the potentially expensive work done by Beautiful Soup into a
function that is called by the alter_content function, so that we can
cache the result.  Saves a significant portion of the runtime of
loading of all of our /help/ and /api/ documentation pages (e.g. 12ms
for /api).

Fixes #11088.

Tweaked by tabbott to use the URL path as the cache key, clean up
argument structure, and use a clearer name for the function.
2019-01-28 15:21:52 -08:00
Matthew Wegner 370cf1a2cb import: Normalize Slackbot String Comparison.
In very old Slack workspaces, slackbot can appear as "Slackbot", and
the import script only checks for "slackbot" (case sensitive).  This
breaks the import--it throws the assert that immediately follows the
test.  I don't know how common this is, but it definitely affected our
import.

The simple fix is to compare against a lowercased-version of the
user's full name.
2019-01-28 14:59:41 -08:00
Steve Howell f2b26b9bcc integration tests: Add check for new messages.
This commit should make it a bit easier to debug
integrations that silently don't send any messages.
2019-01-28 14:56:42 -08:00
Tim Abbott 6994fad237 portico: Specifically mention open source projects in /for/open-source. 2019-01-28 14:50:20 -08:00
Rohitt Vashishtha ff90c0101c bugdown: Process word boundaries properly in realm_filters.
Earlier, our realm filters didn't render for languages that do not
use spaces (eg: Japanese) since we used to check for the presence
of an actual space character. This commit replaces that logic with
a complex scheme to detect word boundaries.

Also, we convert the RealmFilterPattern to subclass InlineProcessor
and make use of the new no-op feature in py-markdown 3.0.1 where we
can tell py-markdown that our pattern didn't find a match despite
the initial regex getting matched.

Fixes #9883.
2019-01-28 14:48:15 -08:00
Steve Howell ad071ced47 bugdown: Avoid recomputing the stream-link regex. 2019-01-28 13:12:37 -08:00
Rohitt Vashishtha 2dc447d707 bugdown: List py-markdown 3.0.1 features that we do not use.
Tweaked by tabbott to extend the documentation.
2019-01-28 13:12:37 -08:00
Rohitt Vashishtha 434094e599 bugdown: Restructure Bugdown to extend Markdown from being an extension.
Since we are building our parser from scratch now:

1. We have control over which proccessor goes at what priority number.
   Thus, we have also shifted the deprecated `.add()` calls to use the
   new `.register()` calls with explicit priorities, but maintaining
   the original order that the old method generated.

2. We do not have to remove the processors added by py-markdown that
   we do not use in Zulip; we explicitly add only the processors we
   do require.

3. We can cluster the building of each type of parser in one place,
   and in the order they need to be so that when we register them,
   there is no need to sort the list. This also makes for a huge
   improvement in the readability of the code, as all the components
   of each type are registered in the same function.

These are significant performance improvements, because we save on
calls to `str.startswith` in `.add()`, all the resources taken to
generate the default to-be-removed processors and the time taken to
sort the list of processors.

Following are the profiling results for the changes made. Here, we
build 10 engines one after the other and note the time taken to build
each of them. 1st pass represents the state after this commit and 2nd
pass represent the state after some regex modifications in the commits
that follow by Steve Howell. All times are in microseconds.

| nth Engine | Old Time | 1st Pass | 2nd Pass |
| ---------- | -------- | -------- | -------- |
|          1 |  92117.0 |  81775.0 |  76710.0 |
|          2 |   1254.0 |    558.0 |    341.0 |
|          3 |   1170.0 |    472.0 |    305.0 |
|          4 |   1155.0 |    519.0 |    301.0 |
|          5 |   1170.0 |    546.0 |    326.0 |
|          6 |   1271.0 |    609.0 |    416.0 |
|          7 |   1125.0 |    459.0 |    299.0 |
|          8 |   1146.0 |    476.0 |    390.0 |
|          9 |   1274.0 |    446.0 |    301.0 |
|         10 |   1135.0 |    451.0 |    297.0 |
2019-01-28 13:12:37 -08:00
Rohitt Vashishtha 9f2c52c86e bugdown: Rename variables regex to REGEX for importing regex module. 2019-01-28 12:00:58 -08:00
Steve Howell 3b7d899532 bugdown: Use CompiledPattern in AtomicLinkPattern.
We avoid re-computing the regex string here, and we
also avoid re-compiling the regex itself.

I decided to put the "one_time" decorator in the
bugdown file itself, just to reduce friction in
folks reading the "buyer beware" comments.

Unfortunately, we can't use this for the
get_web_link_regex() function due to testing concerns,
so that continues to do an inelegant cache-with-global-var
scheme.
2019-01-28 11:58:47 -08:00
Steve Howell eea711a805 bugdown: Flatten get_web_link_regex().
We use early-exit to flatten the code.

I also tweaked the comments a bit based on some recent
profile findings.  (e.g. reading the file isn't actually
a big bottleneck, it's more the regex itself)
2019-01-28 11:58:46 -08:00
Steve Howell 852756aeb3 bugdown: Eliminate LinkPattern class.
The only code that used LinkPattern was AtomicLinkPattern.

We just move the helper method into AtomicLinkPattern.
2019-01-28 11:58:05 -08:00
Steve Howell 77446a710c bugdown: Optimize CompiledPattern.
We don't need the hacky step of passing in a blank
regex to the superclass's __init__ function.

All we need to do is assign `md` to `self.md`.
2019-01-28 11:57:28 -08:00
Steve Howell 025df33d7a bugdown: Rename VerbosePattern to CompiledPattern.
This class only requires a compiled regex--it's up
to the callers how they compile it (verbose or
otherwise).
2019-01-28 11:56:01 -08:00
Eeshan Garg 681f07f98f check_message: Refactor stream validation into separate function. 2019-01-28 11:48:17 -08:00
Eeshan Garg 4cb9a472e6 addressee: Add helper to check for missing or empty topic. 2019-01-28 11:42:26 -08:00
Steve Howell 17a9f20f76 node tests: Use zjquery in narrow.js.
Instead of custom stubs, we now use zjquery.  We also
limit a couple checks to the first call to
`show_empty_narrow_messages`, since it's the same
logic every time.
2019-01-28 07:08:20 -08:00
ss62171 63702d50e6 poll widget: Fix duplicate options bug in poll.
This is a frontend-only change to make it difficult to accidentally
create duplicate options in the poll widget menu.
2019-01-26 15:48:28 -08:00
Tim Abbott df436a55b4 scripts: Recommend apt update after enabling universe.
One needs to manually do an apt update after add-apt-repository, or it
won't actually work.
2019-01-26 12:29:05 -08:00
Vishnu Ks 5fec84221e stats: Make all time active users graph default.
We now bill by total users, whereas we used to
bill by 15 day actives.
2019-01-26 12:19:59 -08:00
Steve Howell 1228e541d7 dispatch test: Restore 100% line coverage (user_status).
We now have 100% coverage on server_events_dispatch.js.

The last piece were was covering "status_text" updates
in the "user_status" event.
2019-01-26 10:48:57 -08:00
Steve Howell 1279249a09 dispatch tests: Add tests for removing starred flags. 2019-01-26 10:48:57 -08:00
Steve Howell f7cf759740 dispatch tests: Stop stubbing message_store.
I am trying to use somewhat more realistic data
for data classes in the dispatch tests.  (We still
want to mostly stub UI stuff.)
2019-01-26 10:48:57 -08:00
Steve Howell 4480846aa5 node tests: Do some zrequires higher in dispatch tests. 2019-01-25 16:53:51 -08:00
Steve Howell 17c906d6bb user status: Surface status in the buddy list.
When you hover over a user that has set a user
status, we now show something like "out to lunch."

You can test this in the console by doing:

    user_status.server_update({status_text: 'out to lunch'})

And then hover over your name in the buddy list.
2019-01-25 16:53:51 -08:00
Steve Howell 6de77d30eb node tests: Test starred message updates. 2019-01-25 16:53:51 -08:00
Steve Howell afe575f02e node tests: Test logo updates. 2019-01-25 16:53:51 -08:00
Steve Howell dfd52ebe1f node tests: Use actual people in dispatch tests.
The stubs here were kind of unnecessary, as the
real people module is lightweight and data setup
is pretty easy.

In passing I also removed the unnecessary `sed`
abbreviation.
2019-01-25 16:53:51 -08:00
Steve Howell 49b16b5b01 activity: Rename function to redraw_user().
The name `insert_user_into_list` is sort of misleading,
since we are often just redrawing the user's existing
item in the buddy list.

I chose `redraw_user` over `update_user` to emphasize
that we're just going to redraw it with whatever data
has been updated by the callers.
2019-01-25 16:53:51 -08:00
kunal-mohta b89769420a stream_color: Extend subscribe API to support color attribute.
This fixes an annoying bug where clicking to subscribe to a stream
would change the color shown in the "manage streams" UI immediately
after you click.

Fixes #11072.
2019-01-25 16:44:59 -08:00
Vishnu Ks ce1db751d0 billing: Make tabs work with back forward buttons. 2019-01-25 16:40:46 -08:00
Vishnu Ks 537b7612cb billing: Remove hash when reloading upgrade page. 2019-01-25 16:40:46 -08:00
Pragati Agrawal 1e811b42ec org settings: Add realm level setting for missed message content in email.
This adds a setting under "Notification" section of
"Organization settings" tab, which enables Organization administrator to
control whether the missed message emails include the message content or
not.

Fixes: #11123.
2019-01-25 14:34:10 -08:00
Vishnu Ks bec875a9af import realm: Use processes for resizing avatar images.
This should significantly improve the data import performance when
importing large open source realms from Slack.

Fixes #11009.
2019-01-25 12:37:12 -08:00
Puneeth Chaganti 33a322baa2 paste: Hook up the paste handler to the message edit form.
This ensures that any nice pasting features we have in the compose box
also apply to message editing.
2019-01-25 11:40:33 -08:00
Puneeth Chaganti 9c377a05f3 paste: More robust check to verify if clipboard data is an image.
The commit f863a9b567 had modified
jquery.filedrop's paste method to exit early if any of the items in the
clipboardData is of the string kind. The early exit was added to prevent pasting
an image thumbnail for text copied from software like MS Word, instead of
pasting the actual copied text content. When copying an image in a (modern?)
Browser, though, the clipboard seems to contain a html `img` tag item, along
with the actual image file. This resulted in pastes being broken.

This commit modifies the condition checked for the early exit. We now actually
look at the html content in the clipboard to see if it is an `img` tag, in which
case we upload the image, instead of exiting early.

Closes #7130.
2019-01-25 11:40:33 -08:00
Priyanshu Singh 8cf5810fc0 docs: Fix a typo in gsoc-ideas.md. 2019-01-25 11:19:18 -08:00
TharunThomas8 2ebd3f244d left sidebar: Fix scrollbar sticking up few pixels above container.
This is likely not the "right" fix in that it involved a negative
margin, but this does eliminate an annoying visual glitch where the
scrollbar overflows above its container in the left sidebar, without
creating other apparent problems.

Fixes #8731.
2019-01-25 11:17:06 -08:00
Mohit Gupta 42d886a6d5 messages: Add loading spinner for deleting messages.
User was able to click delete button multiple time which could cause
multiple delete requests. This commit disables and hides the delete
message button after the first click and shows a spinner until http
the delete request responds.

Also adds a casperjs test to ensure that spinner becomes visible and
delete button becomes invisible after clicking on delete button for
first time and hides spinner and show delete buttton when message is
deleted.

Fixes: #11219.
2019-01-25 11:07:02 -08:00
Mohit Gupta c89356fabf messages: Fix 500 error with duplicate delete message requests.
Multiple delete message requests for the same message sometimes caused
a 500 error.  This happened via the normal IntegrityError being thrown
by delete message/archiving code.

This was manually reproduced by adding latency in function
move_messages_to_archive() in retention.py and
delete_message_backend() in views.py.  This addresses the problem by
adding code to handle the exception and throw JsonableError to convert
500 to 400 errors, with an automated test.
2019-01-25 11:07:02 -08:00
Steve Howell c172001f22 minor: Finish last fix.
See the prior commit.  I missed another info -> status_text
thing.
2019-01-25 07:55:37 -08:00
Steve Howell 45d07e8472 user status: Fix stray "info" reference.
This fixes a section of code that hasn't really
been turned on yet.  We decided to rename
"info" to "status_text", and I apparently missed
this.  We don't have any UI to set these yet,
so it was a harmless bug.

I'll try to get some better test coverage on this
when I tweak the buddy list to show user status.
2019-01-25 07:45:00 -08:00
Tim Abbott a64c434b9e topics: Fix messages_for_topic to use iexact matching.
This fixes the archive codebase to use our covention of using
case-insensitive matching for message topics.
2019-01-24 17:55:15 -08:00
Eeshan Garg 5057406f16 webhooks/clubhouse: Extract story labels from actions.
Apparently, Clubhouse has two different payloads for story label
changes, one where the label name lives inside the "references"
object, and the other where it lives inside the "actions" object.
Their webhook API is still in beta, so this could just be a bug.
Anyhow, we should support both.
2019-01-24 20:08:59 -03:30
Vaibhav 746870df65 stream settings: Hide creating admin-only streams in members UI.
This commit takes away the ability for non-admin members to create
streams where only admins can post messages by hiding the option from
them.

Fixes #11290.
2019-01-24 10:51:49 -08:00
Vaibhav 268c25e95f streams: Prevent members from creating admin-only streams.
This a check on server side to verify whether the user sending request
to create stream where only admins can post is an admin or not; Raises
a JsonableError when the user is not the realm admin.
2019-01-24 10:51:45 -08:00
Eeshan Garg 2b81d66537 webhooks/clubhouse: Add support for GitHub handler events.
These events are triggered when one has Clubhouse's GitHub
integration and custom event handlers set up.
2019-01-24 10:01:17 -08:00
Steve Howell 513d666ba1 user status: Add frontend data layer for "status_text".
This sets up the frontend to handle "status_text" data like
"out to lunch" or "in a meeting".
2019-01-24 09:46:35 -08:00
Steve Howell 0ef5d1f9c8 user status: Add "status_text" support in the backend.
You can now pass in an info field with a value
like "out to lunch" to the /users/me/status,
and the server will include that in its outbound
events.

The semantics here are that both "away" and
"status_text" have to have defined values in order
to cause changes.  You can omit the keys or
pass in None when values don't change.

The way you clear info is to pass the empty
string.

We also change page_params to have a dictionary
called "user_status" instead of a set of user
ids.  This requires a few small changes on the
frontend.  (We will add "status_text" support in
subsequent commits; the changes here just keep
the "away" feature working correctly.)
2019-01-24 09:46:35 -08:00
Steve Howell ac861f2b7d user status: Restructure code.
We now have single function that handle both away
and not-away.

This refactoring sets us up to piggyback "info" more
easily onto status updates.

The only thing that changes here is that we don't
delete database rows any more when users revoke
their away status.  Instead we just set the status
to NORMAL.
2019-01-24 09:46:35 -08:00