Commit Graph

32390 Commits

Author SHA1 Message Date
Mateusz Mandera 3fba4cfbba retention: Add restore_messages management command. 2019-06-26 12:21:46 -07:00
Mateusz Mandera 3ac11a3fc5 retention: Use ON CONFLICT DO UPDATE to handle re-archiving properly.
When archiving Messages, we stop relying on LEFT JOIN ... IS NULL to
avoid duplicates when INSERTing. Instead we use ON CONFLICT DO UPDATE
(added in postgresql 9.5) to, in case of archiving a Message that
already has a corresponding archived objects (this happens if a Message
gets archived, restored and then archived again), re-assign the existing
ArchivedMessage to the new transaction.

This also allows us to fix test_archiving_messages_second_time, which
was temporarily disable a few commits before.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 7b2b4435ed retention: Combine run_message_batch_query and run_archiving_in_chunks.
We combine run_message_batch_query and run_archiving_in_chunks
functions, which makes the code simpler and more readable - we get rid
of hacky generator usage, for example.
In the process, move_expired_messages_* functions are adjusted, and now
they archive Messages as well as their related objects.
Appropriate adjustments in reaction to this are made in the main
archiving functions which call move_expired_messages_* (they no longer
need to call move_related_objects_to_archive).
2019-06-26 12:05:59 -07:00
Mateusz Mandera 6e46c6d752 retention: Add functions for restoring archived data.
Functions for restoring archived data are added and existing tests are
expanded to restore data they archived and check correctness.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 9acd3b0f46 retention: Rewrite move_messages_to_archive to use existing functions.
Instead of having a bunch of custom code in the function, we make it use
run_message_batch_query and run_archiving_in_chunks to do the necessary
operations in a consistent way, using the same codepaths as the rest of
the archiving system.
This breaks test_archiving_messages_second_time temporarily, but we will
fix it and re-enable the test in the next commits, where we'll address
various other issues with re-archiving of messages.

We also remove the @transaction.atomic wrapper, because atomicity is
handled by the logic inside run_archiving_in_chunks.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 5b20317379 retention: Add __str__ method to ArchiveTransaction. 2019-06-26 12:05:59 -07:00
Mateusz Mandera c869ea8e1e test_retention: Factor out a class with shared helper functions. 2019-06-26 12:05:59 -07:00
Mateusz Mandera 7fc48f8b93 test_retention: Check if messages get deleted when archiving.
We add additional checks in _verify_archive_data to make sure the
archived Messages and UserMessages are deleted from their normal tables.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 80b834dd1b retention: Update move_rows() function code.
We make minor changes to the move_rows() function to allow its use in
the code for restoring from the archive.
2019-06-26 12:05:59 -07:00
Mateusz Mandera e3fe66a084 retention: Set savepoint=False on atomic wrapper on move_rows().
Savepoints create unnecessary overhead, and there's no benefit from
them, with the way we use this function.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 5d8d5910a8 retention: Log archive_transaction id and information. 2019-06-26 12:05:59 -07:00
Mateusz Mandera a2cce62c1c retention: Use new ArchiveTransaction model.
We add a new model, ArchiveTransaction, to tie archived objects together
in a coherent way, according to the batches in which they are archived.
This enables making a better system for restoring from archive, and it
seems just more sensible to tie the archived objects in this way, rather
the somewhat vague setting of archive_timestamp to each object using
timezone_now().
2019-06-26 12:05:59 -07:00
Anders Kaseorg e8d49330f2 templates: Replace logic helpers with sensible logic helpers.
`if_not_a_or_b_and_not_c`?  Really?

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 12:03:53 -07:00
Tim Abbott aea1279e8c puppet: Remove trusty configuration for static_asset_compiler.
Trusty is desupported.
2019-06-26 11:32:06 -07:00
Tim Abbott 8fbd965ab5 puppet: Remove legacy pgtune related configuration for trusty.
Since we no longer support Ubuntu Trusty, we no longer need this
backwards-compatibility cruft (which we only kept around to avoid
randomizing configuration for existing systems).
2019-06-26 11:32:06 -07:00
Anders Kaseorg 33c941407b puppet: Remove legacy unauthenticated local uploads backend.
This was only used in Ubuntu 14.04 Trusty.

Removing this also finally lets us simplify our security model
discussion of uploaded files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:31:46 -07:00
Anders Kaseorg 6112d020f4 migrate, create_large_indexes: Use CREATE INDEX IF NOT EXISTS.
We no longer support Postgres versions missing this syntax.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:01:30 -07:00
Anders Kaseorg bfa82a70c8 ci: Remove Ubuntu 14.04 (trusty) from CI.
Tweaked by tabbott to improve our CI docs discussion of these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:01:13 -07:00
Anders Kaseorg b4204e7cfd Remove legacy (pre-Vagrant) Docker development environment setup.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 10:58:02 -07:00
Hemanth V. Alluri cd9c1bfb9d devtools: Update the send_all feature of the integrations tool for headers.
Now that we have a system for storing HTTP headers for each integration, we
should fix the send_all button. Previously, it used the same user entered
custom HTTP header (from the GUI) for all of the fixtures, but now we
automatically determine the header with the new system instead.
2019-06-26 10:35:16 -07:00
Hemanth V. Alluri ef52aa0fc1 webhooks: Eliminate the usage of a headers.py file.
For storing HTTP headers as a function of fixture name, previously
we required that the fixture_to_headers method should reside in a
separate module called headers.py.

However, as in many cases, this method will only take a few lines,
we decided to move this function into the view.py file of the
integration instead of requiring a whole new file called headers.py

This commit introduces the small change in the system architecture,
migrates the GitHub integration, and updates the docs accordingly.
2019-06-26 10:35:14 -07:00
Hemanth V. Alluri 4691028097 webhooks: Extract the '__' method of mapping to headers.
In the GitHub integration we established that for many integrations,
we can directly map the fixture filename to the set of required
headers and by following a simple naming convention we can greatly
ease the logic involved in fixture_to_headers method required .

So to prevent the need for duplicating the logic used by the GitHub
integration, we created a method called `get_http_headers_from_filename`
which will take the name of the HTTP header (key) and then return a
corresponding method (in a decorator-like fashion) which could then be
equated to fixture_to_headers in headers.py.

The GitHub integration was modified to use this method and the docs
were updated to suggest using this when possible.
2019-06-26 10:32:34 -07:00
Tim Abbott 271319fb13 puppet: Fix hacky release test for whether we're in EC2.
The result is still a bit hacky, but guaranteed to be correct if we
adjust the OS version of our systems, which we of course will do over
time.
2019-06-25 22:19:04 -07:00
Tim Abbott 8d8cfb314b puppet: Remove zulip_ops configuration for trusty.
There are no longer any zulip_ops systems using trusty.
2019-06-25 22:09:06 -07:00
Pragati Agrawal e42abc2396 org_settings: Optimize data collected by `populate_data_for_request`.
With the help of `check_property_changed` function now we collect the data
whose values are changed from the current one. Currently this optimizes
only for those elements whose values are collected by
`populate_data_for_request` function i.e. it doesn't optimize data
collected by `get_complete_data_for_subsection`.
2019-06-25 16:57:59 -07:00
Pragati Agrawal bdbf63ed27 org_settings: Move `populate_data_for_request` down.
This is preliminary commit which moves `populate_data_for_request` function
down after the definition of all functions with which it will interact in
the future.
2019-06-25 16:57:59 -07:00
Pragati Agrawal 81492362d7 org_settings: Refactor `populate_data_for_request`.
This is a preliminary commit which refactors `populate_data_for_request`
function, now this function traverse on all "property elements" of a given
subsection, but get the data only of those properties which have
`setting-widget-type` data attribute. Therefore, it doesn't change the
functionality of this function and overall changes don't make any
difference. In upcoming commits, we're going to use `input_elem` as an
argument to `check_property_changed` function, so that only those elements
whose values are changed are sent to the backend.
2019-06-25 16:57:59 -07:00
Pragati Agrawal d200d662d7 org_settings: Move `get_subsection_property_elements` from `build_page`.
This moves `get_subsection_property_elements` out from the local context of
`settings_org.build_page` function, as it was unnecessarily initialized at
the time of page setup.
2019-06-25 16:57:59 -07:00
Pragati Agrawal 15ff8fb73f settings: Fix sort order display settings.
This fixes the mis-alphabetized `fluid_layout_width` at few places in
the codebase, along with that it also fixes sorting order of
`property_types` dictionary in models.py and few model fields of
`UserProfile` model class.
2019-06-25 16:57:06 -07:00
Tim Abbott 216267e279 Revert "message_edit: Change scroll calculations based on `.message_content`."
This reverts commit edc5337d53.

This somehow broke the 08-edit casper test.
2019-06-24 23:29:28 -07:00
Anders Kaseorg bac63f5a3e documentation_crawler: Reimplement crawl_with_status so it wraps crawl.
This way we inherit more of the upstream command’s behavior.
Importantly, this means we pass everything in `opts.spargs` to the
spider, not just `opts.spargs.skip_external`.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-24 23:20:17 -07:00
Anders Kaseorg a37fa74dce templates: Fix new low-hanging HTML validation problems.
"http://localhost:9981/api/update-message-flags":318.16-319.34: error: “th” start tag in table body.
"http://localhost:9981/api/update-message-flags":344.35-344.38: error: Stray end tag “a”.
"http://localhost:9981/api/incoming-webhooks-walkthrough":412.4-412.77: error: An “img” element must have an “alt” attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-24 23:20:17 -07:00
Eeshan Garg daea9a543b webhooks/bitbucket2: Account for missing username in PR payloads.
Certain PR payloads do not have a username for the reviewer, in
such cases, we just return the reviewer's nickname or display
name.
2019-06-24 23:19:06 -07:00
Tim Abbott f9aef15e72 message_edit: Clean up comments around scroll position. 2019-06-24 22:53:57 -07:00
Vaibhav edc5337d53 message_edit: Change scroll calculations based on `.message_content`.
This concerns the part where the message viewport is scrolled in
order to keep the message content and edit box textarea at the same
vertical position.

Earlier calculations involved use of `.message_top_line`. Similar
adjustments can be made using only the message_content (which also
makes the calculation simpler).
2019-06-24 22:49:42 -07:00
Tim Abbott 6763e77ee2 api docs: Fix syntax for get-user-groups.
This apparently had some sort of rebase conflict issue with the
migration to the nicer syntax for tabs.
2019-06-24 22:24:46 -07:00
Pragati Agrawal 2e97b03b2b css: Use SCSS nesting in `settings.scss` for `.new-style`. 2019-06-24 22:14:13 -07:00
Pragati Agrawal b30ae0b7f5 css: Remove redundant `#settings` from `settings.scss`.
This commit removes `settings` element from setiings.scss since now no
such element exists.
2019-06-24 22:12:42 -07:00
Pragati Agrawal 45677990fd css: Remove redundant `#organization` from `settings.scss`.
This commit removes the redundant `#organization` from settings.scss as
organization id no longer exists.
2019-06-24 22:12:42 -07:00
Vaibhav fa85b92eda populate_db: Add MD content to stream descriptions and user bio.
This adds a 'bold' element to `all` stream and 'code' element to
`test` stream in populate_db. Also changes the bio for Hamlet to
include a list.
2019-06-24 22:10:24 -07:00
Vaibhav 6e05131e9a markdown: Add `.rendered_markdown` to elements in with MD content.
These elements include:
* Stream description in the subscription overlay
  * sidebar and
  * stream settings
* Custom profile fields with rendered MD content
2019-06-24 22:10:24 -07:00
Vaibhav 336ea98dd9 css: Remove buggy white-space style from profile modal content.
This is to make the user profile settings rendered content in accordance
to the message box styling.
2019-06-24 22:10:24 -07:00
Anders Kaseorg 0f30c8c4ba dev-vagrant-docker: Upgrade docker-systemctl-replacement.
This fixes the 45 second boot delay bug.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-24 22:06:32 -07:00
Yago González 4dc206ad2e api docs: Document PATCH /settings/notifications.
With significant tweaks by tabbott after rebasing.
2019-06-24 18:42:44 -07:00
Thomas Ip 80f972d42c webpack: Use correct types in config.
Cache-loader is used as an item in the use member so the correct
type should be RuleSetUseItem not RuleSetRule.
See: DefinitelyTyped/DefinitelyTyped/webpack/index.d.ts#L498
2019-06-24 17:58:51 -07:00
Thomas Ip 637b2b784a dependencies: Upgrade mini-css-extract-plugin to 0.6.0 and use HMR.
Given mini-css-extract-plugin can now do hot module replacement,
this commit also removed css-hot-loader. Not upgrading to 0.7.0
as that cause webpack to crash.
2019-06-24 17:58:51 -07:00
Thomas Ip 413c3066f0 dependencies: Upgrade to ts-loader 6.0.2. 2019-06-24 17:58:27 -07:00
Thomas Ip 1c99cb320e dependencies: Upgrade swagger-parser to 6.0.5.
4.0.0 changed how options worked but we didn't use that so no
changes needed.
2019-06-24 17:58:27 -07:00
Thomas Ip 6f2a909e22 dependencies: Upgrade style-lint to 10.0.1. 2019-06-24 17:58:27 -07:00
Thomas Ip 308d7bb8f5 dependencies: Upgrade cache-loader to 4.0.0. 2019-06-24 17:58:27 -07:00