Commit Graph

8248 Commits

Author SHA1 Message Date
Tim Abbott 3b36532407 storage: Fix hash-renaming of various non-CSS/JS file types.
None of the file types here are actually processed by our static asset
pipeline in a way that would result in the hash-named versions of the
files (stored in staticfiles.json) being accessed.  (If they were,
we'd be using something like `render_bundle` to access their paths).

So, we should not be generating/shipping these hash-named versions of
image, audio, and locale files.

This change decreases the size of a Zulip release tarball from 153MB
to 93MB, by removing all of the duplicated copies of various asset
files.

This may also help with #10038, but I'm not marking it as completing
that issue yet, because part of #10038 is that the non-hash-named
image files in prod-static/generated/emoji do not seem to have been
properly overwritten on upgrade, and it's unclear why.

Fixes #5971.
2018-07-24 09:42:25 -07:00
Tim Abbott e74c37a129 migrations: Convert migration 0041 to not import from zerver.
This should help avoid problems when upgrading from very old versions of Zulip.
2018-07-24 09:00:14 -07:00
amanagr 284259d1c3 tests: Remove redundant test for message 'flags' in test_events.py. 2018-07-23 23:10:07 -07:00
Tim Abbott 08d890e671 test_events: Use check_dict_only for stream_weekly_traffic. 2018-07-23 17:37:24 -07:00
Rishi Gupta e37b5dce79 stream settings: Fix brand new streams showing 0 weekly traffic.
A stream created in the last few hours likely won't be in StreamCount
(since that gets updated once a day), and hence won't be in the
recent_traffic dict.

However, get_average_weekly_stream_traffic should be None in this case,
not 0.
2018-07-23 16:43:15 -07:00
Rishi Gupta 0573c17819 stream settings: Set weekly traffic to None instead of -1 for new streams.
The only place this gets used seems to be subscription.handlebars, where a
value of None (or null) is probably less broken-seeming than -1 anyway.
2018-07-23 16:43:15 -07:00
Rishi Gupta b75aa8e35a stream settings: Fix age calculation in is_old_stream.
Match the calculation in get_average_weekly_stream_traffic.
Regression introduced in dc7cfd3.

Fixes #10002
2018-07-23 16:43:15 -07:00
Joshua Pan a9d9919648 Add delivery_email field to page_params.
This allows the frontend to use the `delivery_email`
field for display use.
2018-07-23 15:38:41 -07:00
Harshit Bansal c0b0fb7cce emoji: Move `EMOTICON_CONVERSIONS` mapping to build_emoji infra.
This commit closes a long pending issue which involved moving the
`EMOTICON_CONVERSION` mapping to build_emoji infrastructure so
that there is only one source of truth. This was pending from the
time when this feature was implemented.
2018-07-23 12:35:08 -07:00
Tim Abbott 2918cc92f3 search: Duplicate HTML template content for search area.
This is preparation for being able to work on the search pills feature
without making any user-facing changes until we're ready to enable it.
2018-07-23 11:29:10 -07:00
Tim Abbott 8dca8f1087 search: Add setting to control whether search pills are enabled.
This setting isn't intended to exist long term, but instead to make it
possible to merge our search pills code before we're ready to cut over
production environments to use it.
2018-07-23 11:29:10 -07:00
Rhea Parekh 8803ac4af8 import: Fix BotStorageData and BotConfigData import.
The function 'update_model_ids' should be used on
the models BotStorageData and BotConfigData.
It is wrongly added here for UserGroup model.

Also the sequence name for BotStorageData and
BotConfigData is 'zerver_botuserstatedata_id_seq' and
'zerver_botuserconfigdata_id_seq' respectively, which
should be specifically mentioned in the function
'allocate_ids'.

This fixes some nondeterministic test failures.
2018-07-23 11:24:17 -07:00
Tim Abbott 839300d781 import: Fix typo in re_map_foreign_keys_many_to_many.
This was introduced when I rebased together the two implementations of
this function.
2018-07-23 10:19:04 -07:00
Rhea Parekh a656257e6b gitter import: Process user mentions.
The gitter mentions are in the format '@usermention'
and the mentions are included in the export data as:

"mentions": [
   {
    "screenName": "usermention",
    "userId": "54d7876c15522ed4b3dbbefb",
    "userIds": []
   }]

We extract this data and map this mention to @**usermention**
for Zulip.
2018-07-23 08:36:30 -07:00
Rhea Parekh de13704d0b gitter import: Write messages batch-wise.
Messages can be bulky, and storing them in a single
data structure can cause a memory error.

In this commit, the messages are written to a file
batch-wise, thus avoiding the memory error.

Similar to commit 6b7b6b38ad
2018-07-23 08:36:30 -07:00
Rhea Parekh a0034bfb6e gitter import: Add basic tests. 2018-07-23 08:36:30 -07:00
Rhea Parekh 36aed2c08d gitter import: Add management command. 2018-07-23 08:36:30 -07:00
Rhea Parekh 3a6921e54b gitter import: Add gitter data conversion script. 2018-07-23 08:36:30 -07:00
Tim Abbott db1260fb93 export: Clean up comments on why tables are not currently exported. 2018-07-23 08:28:20 -07:00
Rhea Parekh c42d6b6983 export: Remove 'zerver_pushdevicetoken' from the to be imported list.
PushDevicetoken is automatically created when a user logs
in a server from mobile. This shouldn't be imported.
2018-07-23 08:21:00 -07:00
Rhea Parekh fe299277f3 import: Import BotStorageData and BotConfigData. 2018-07-23 08:21:00 -07:00
Rhea Parekh f01ff28e03 export: Export BotStorageData and BotConfigData. 2018-07-23 08:21:00 -07:00
Rhea Parekh 2978e025df import: Import UserGroup. 2018-07-23 08:21:00 -07:00
Rhea Parekh 98a7762a51 export: Export user groups. 2018-07-23 08:20:58 -07:00
Rhea Parekh bc2307108d import: Import Service. 2018-07-23 08:20:58 -07:00
Rhea Parekh 0fcf6d9a40 export: Export Service. 2018-07-23 08:20:58 -07:00
Rhea Parekh f444e5b628 import: Import MutedTopic. 2018-07-23 08:20:58 -07:00
Rhea Parekh 6eab6446fc export: Export MutedTopic. 2018-07-23 08:20:58 -07:00
Rhea Parekh 091d101e7d import: Import UserHotspot. 2018-07-23 08:20:58 -07:00
Rhea Parekh 8897e187c0 export: Export UserHotspot. 2018-07-23 08:20:58 -07:00
Rhea Parekh 6f7b7e143f import: Map user IDs for custom profile field of type 'USER'.
The CustomProfileField object which has the `field_type`
`USER` needs to be updated with the new user IDs.
2018-07-23 08:19:04 -07:00
Rhea Parekh 3879e345d9 import: Add function to re-map foreign keys for ManyToMany fields.
This will be used while for any ManyToMany field which
is being imported.
We add an internal function which takes in the old ID list
of the ManyToMany field and return the new updated ID list.
2018-07-23 08:19:04 -07:00
Tim Abbott bdc95b5d72 slack webhooks: Eliminate unnecessary use of Service model.
The user ID of the bot user is a much more useful value to be sending
to the third-party API here.
2018-07-23 08:12:08 -07:00
Vishnu Ks 2b28042ddf models: Rename remaining audit log event types to past tense.
This makes the realm audit event type log entries more consistent.
2018-07-22 20:00:28 -07:00
Vishnu Ks 28167585b1 models: Rename several RealmAuditlog event types to past tense.
These event types haven't been used in production yet, and thus are
save to change without a migration.
2018-07-22 20:00:05 -07:00
amanagr 2d5ef27874 message-model: Add `zulipterminal` to sent_by_human clients.
Tweaked by tabbott to also more fully document this feature.
2018-07-22 12:32:55 -07:00
Anders Kaseorg a0293e8a86 zerver/lib/import_realm.py: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 195cc78470 zerver/lib/cache.py: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Steve Howell 5d0a113634 minor: Add assertion to assert_realm_values() helper. 2018-07-19 12:07:11 -04:00
Aditya Bansal 98a4e87e1d thumbor: Complete implementation of thumbnailing.
Various pieces of our thumbor-based thumbnailing system were already
merged; this adds the remaining pieces required for it to work:

* a THUMBOR_URL Django setting that controls whether thumbor is
  enabled on the Zulip server (and if so, where thumbor is hosted).

* Replaces the overly complicated prototype cryptography logic

* Adds a /thumbnail endpoint (supported both on web and mobile) for
  accessing thumbnails in messages, designed to support hosting both
  external URLs as well as uploaded files (and applying Zulip's
  security model for access to thumbnails of uploaded files).

* Modifies bugdown to, when THUMBOR_URL is set, render images with the
  `src` attribute pointing /thumbnail (to provide a small thumbnail
  for the image), along with adding a "data-original" attribute that
  can be used to access the "original/full" size version of the image.

There are a few things that don't work quite yet:
* The S3 backend support is incomplete and doesn't work yet.
* The error pages for unauthorized access are ugly.
* We might want to rename data-original and /thumbnail?size=original
  to use some other name, like "full", that better reflects the fact
  that we're potentially not serving the original image URL.
2018-07-15 00:39:41 +05:30
Sarah de947445ca event_queue: Add stream_email_notify.
This adds support to the event queue system for triggering
missed-message notifications (whether push or email) to support the
stream push notifications feature.
2018-07-14 12:19:33 +05:30
Sarah 73f5be65e8 zerver/lib/notifications: Add stream email notifications.
This modifies the logic for formatting outgoing missed-message emails
to support the upcoming stream email notifications feature (providing
a new format for the subject, etc.).
2018-07-14 12:19:33 +05:30
Tim Abbott 47df8c0c6e notifications: Elimininate now-unnecessary database query.
Because we're passing through the trigger for notifications to
do_send_missedmessage_events_reply_in_zulip, we don't need to go back
to the database to determine which messages actually mentioned the
user.
2018-07-14 12:19:33 +05:30
Tim Abbott dabf3667d1 notifications: Eliminate unnecessary at_mention context variable.
we already had a variable "mention" that did the exact same thing.
2018-07-14 12:19:33 +05:30
Tim Abbott bc861b4441 notifications: Pass trigger to do_send_missedmessage_events_reply_in_zulip.
This change converts our logic for determining whether the current
user was mentioned in a group of messages from the implicit "if it was
sent to a stream, it's a mention" to the explicit "we actually know
there was a mention in the message".  This is an important
prerequisite for our upcoming feature to support getting email
notifications for streams always (even without a mention).
2018-07-14 12:02:08 +05:30
Tim Abbott 58a7a390c8 event_queue: Call build_offline_notification unconditionally.
Previously, maybe_enqueue_notifications had this very subtle logic,
where it set the notice variable only inside the block for push
notifications, but then also used it inside the block for email
notifications.

This "worked", because previously the conditions for push
notifications were always true if the conditions for email
notifications were, but the code was unnecessarily confusing.  The
only good reason to write it this way is if build_offline_notification
was expensive; in fact, the most expensive thing it does is calling
time.time(), so that reason does not apply here.

This was further confusing, in that in the original logic, we relied
on the fact that push notification code path edited the "notice"
dictionary for further processing.

Instead, we just call it separately and setup the data separately in
each code path.
2018-07-14 12:01:59 +05:30
Tim Abbott a09ebf0551 event_queue: Remove confusing comment about rabbitmq.
Whatever RabbitMQ check this comment used to be next to, it isn't next
to anymore.
2018-07-14 12:00:37 +05:30
Tim Abbott 08ed1b4605 notifications: Set trigger on offline email notifications.
This data will be required for correctly implementing the upcoming
stream_push_notify feature; it also helps support cleaning up the code
for the existing stream mentions logic.
2018-07-14 11:57:28 +05:30
Tim Abbott b0c393cf97 notifications: Format missed_messages argument as a dict.
Because in upcoming commits, we'll want to pass additional per-message
data into do_send_missedmessage_events_reply_in_zulip, we need to
expand the format for how we represent messages to account for that.
2018-07-14 11:34:04 +05:30
Aditya Bansal 2e837b1407 archives: Change the topic/<topic_name> code path to use topics prefix.
We do this for maintaining consistency in naming of the endpoints.
2018-07-14 10:39:35 +05:30