Commit Graph

37908 Commits

Author SHA1 Message Date
Tim Abbott 525b42cecc setup_venv: Require same Python version for virtualenv-clone.
This prevents us cloning a virtualenv in a way that would cause us to
ignore a newly updated Python version on the system.
2020-07-13 13:06:15 -07:00
Aman Agrawal 685ec2a098 hash_reqs: Include python version when generating hash.
Fixes #12868.
We now also include python version in the format
'major.minor.patchlevel', when generating hash for a
requirement file. This was necessary since packages tend to
break on different versions of python, so it is important to
track the version on which the venv was setup.

WARN: This commit will force all zulip venvs to be recreated.
2020-07-13 13:06:15 -07:00
Aman Agrawal 2668829c93 hash_reqs: Use combined package name and version to generate hash.
We were already using packages names along with their versions
to generate hash for the requirement file, as we were passing
the `.txt` files to the hash_reqs file instead of intended `.in` files
for which the functions in this file was originially designed.

Changed the expand_reqs_helper function to adapt for the `.txt` files.
2020-07-13 13:06:15 -07:00
Alex Vandiver 2a14212b27 munin: Add a helper resource definition for munin plugins. 2020-07-13 12:49:28 -07:00
Alex Vandiver 7c7b5fcd6f munin: Deal with spaces in the channel names. 2020-07-13 12:49:28 -07:00
Alex Vandiver eda2c4b8e2 puppet: Split munin-node from munin-server.
No plugins are installed inside the /usr/local/munin/lib this creates
in munin-node, nor are they symlinked into /etc/munin/plugins, so
non-default plugins are added by this.
2020-07-13 12:49:28 -07:00
Alex Vandiver ddc7bb5a45 munin: Fix the path to check_send_receive_time. 2020-07-13 12:49:28 -07:00
Alex Vandiver 8be544e7eb munin: Rename monitoring plugin to use zulip name, not humbug. 2020-07-13 12:49:28 -07:00
Alex Vandiver 1b3560af94 nagios: Stop assuming /api is where zulip client is.
The api/ directory was removed in f9ba3cb60c; as that commit notes, we
use the python-zulip-api module for that, added in 938597c5da.
2020-07-13 12:49:28 -07:00
Alex Vandiver 5ff3025411 upgrade: Add additional comments. 2020-07-13 12:47:49 -07:00
Alex Vandiver 47400cd04b upgrade: Drop unnecessary memcached restart.
The contents in the database are unchanged across the PostgreSQL
restart; as such, there is no reason to invalidate the caches.

This step was inherited from the general operating system upgrade
documentation.  When Python versions change, such as during OS
upgrades, we must ensure that memcached is cleared.  However, the
`do-release-upgrade` process uninstalled and upgraded to a new
memcached, as well as likely restarted the system; a separate step for
OS upgrades to restart memcached is thus unnecessary.
2020-07-13 12:47:04 -07:00
Alex Vandiver 0502b7a8d5 upgrade: Drop the unnecessary step that stops the old cluster.
The initial step in pg_upgradecluster stops the cluster for us; this
removes the somewhat ugly hack we are otherwise forced into.
2020-07-13 12:45:50 -07:00
Alex Vandiver bf0f712c81 upgrade: Use the in-place pg_upgrade, not a full dump/restore.
pg_upgradecluster has two possibilities for `--method`: `dump`, and
`upgrade`.  The former is the default, and does a `pg_dump` of all of
the databases in the old cluster and feeds them into the new cluster.
This is a sure-fire way of getting the same information in both
databases, but may be extremely slow on large databases, and is
guaranteed to fail on servers whose databases take up >50% of their
disk.

The `--method=upgrade` method, by contrast, uses pg_upgrade to copy
the raw database data file over to the new cluster, and then fiddles
with their internal structure as needed by the upgrade to let them be
correct for the new version[1].  This is slightly faster than the
dump/load method, since it skips the serialization step, but still
requires that there be enough space on disk for both old and new
versions at once.  `pg_upgrade` is currently supported for all
versions of PostgreSQL from 8.4 to 12.

Using `pg_upgrade` incurs slightly more risk, but since the it is
widely used by now, using it in the relatively-controlled Zulip server
environment is reasonable.  The expected worst failure is failure to
upgrade, not corruption or data loss.

Additionally passing `--link` uses hardlinks to link the data files
into both the old and new directories simultaneously.  This resolve
both the runtime of the operation, as well as the disk space usage.
The only potential downside to this is that as soon as writes have
occurred on the upgraded cluster, the old cluster can no longer be
started.  Since this tooling intends to remove the old cluster
immediately after the upgrade completes successfully, this is not a
significant drawback.

Switch to using `--method=upgrade --link`.  This technique spits out
two shell scripts which are expected to be run after completion of the
upgrade; one re-analyzes the statistics, the other does an `rm -rf` of
the data where it is still hardlinked in the old cluster.  Extract the
location of these scripts from parsing the `pg_upgradecluster` output;
since the path is not static, we must rely on it being relatively easy
to parse.  The risk of the path changing is lower, and has more
obvious failure modes, than inserting the current contents of these
upgrade steps into the overall `upgrade-postgres`.

[1] https://www.postgresql.org/docs/12/pgupgrade.html
2020-07-13 12:45:50 -07:00
Mateusz Mandera c231d88d9f upgrade: Add management command to fix FTS indexes.
Upgrading the base OS's dictionary files can corrupt our FTS
indexes. We add a command for fixing them.

Fixes #14982.
2020-07-13 12:40:44 -07:00
Priyansh Garg 23fe58f5be
docs: Fix typo in git/using.md. 2020-07-13 12:35:07 -07:00
rht 6b0e354c74 IRC integration: Clarify the doc of requirements.txt installation. 2020-07-13 12:34:20 -07:00
rht 0325d6d18d IRC integration: Update doc to include a cd to the IRC dir. 2020-07-13 12:34:20 -07:00
Ryan Rehman 14c869d803 message send: Update failed message ui on re-send success.
We have logic in place to update the ui for re-sending messages
on recieving the acknowledgement from the server on that API call.

However, if the acknowledgement is recieved through the get events
request before the `on_success` of `resend_message`, the message
gets re-rendered allowing the failed message actions to be clickable.

Now, we update the ".message_failed" ui for both cases. This helps
in preventing the "Trying to get local_id from row that has reified
message id" exception.

Fixes #15351.
2020-07-13 12:29:33 -07:00
Rohitt Vashishtha 37c0577f8a markdown: Assert twitter previews render inside spoilers.
Previously, we rendered the twitter previews outside of a
spoiler block at the end of the message. The commit series
ending with this commit fixes that by inlining twitter
previews instead of appending them all at the end. As a
consequence of the inlining, we have fixed the issue here.
This commit just adds a test to assert that.

Fixes #15518.
2020-07-13 12:24:32 -07:00
Rohitt Vashishtha d3770153a6 markdown: Remove paragraphs that only contain a tweet link.
This is similar to our behavior with image previews, and helps
reduce clutter in the final rendered html.

We add the string 'Tweet: ' to our existing tests so those tests
remain the same.
2020-07-13 12:24:32 -07:00
Rohitt Vashishtha 87e01cd1fa markdown: Inline Twitter previews instead of appending at end.
This commit makes our handling of twitter previews consistent with
how we handle our inline images so that tweets render next to the
paragraph that links to the tweet.
2020-07-13 12:24:32 -07:00
Rohitt Vashishtha a8ab745ee4 markdown: Extract get_inlining_information for link previews.
We decouple the logic of insertion rules for inline links from
image preview logic. Now, we can use this same logic for other
kinds of link previews as well.
2020-07-13 12:24:32 -07:00
Aman Agrawal 55f80a2502 recent_topics: Don't render topic of deactivatedstreams.
Fixes error when trying to render topic of a deactivated stream in
the Recent Topics widget.
2020-07-13 11:22:43 -07:00
Aman Agrawal 46c966576d scss: Fix broken rules. 2020-07-13 10:46:34 +05:30
Mateusz Mandera 57d3ef42b8 puppet: Don't run thumbor services in production.
Fixes #15649.
Currently, no production services use thumbor; so, it makes sense
to not run them in production systems.
2020-07-10 14:22:17 -07:00
Tim Abbott afc0da6979 i18n: Update translation data from Transifex. 2020-07-10 12:45:33 -07:00
Aman Agrawal c32f27a05d hash_util: Show error if url is invalid.
For urls we cannot handle, we inform user via a nice
error message.

See comment in decodeHashComponent for some of the cases it
fixes for us.
2020-07-10 11:01:31 -07:00
Aman Agrawal f97d35ebd2 ui_report: Use fadeOut to hide error.
fadeTo is not a good method to hide elements since it sets
opacity to 0 in which the element still can consume space and
be clickable. We set it's display to None using fadeOut method.

Also, allow this method to be called via ui_report.error.
2020-07-10 11:01:31 -07:00
Anders Kaseorg 7e8280ef50 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-10 00:14:43 -07:00
Alex Vandiver f0f29584aa puppet: Add an arity count ("at least two") to zulipconf function. 2020-07-10 00:14:09 -07:00
Alex Vandiver 8cff27f67d puppet: Pull hosts from zulip.conf, not hardcoded list.
The one complexity is that hosts_fullstack are treated differently, as
they are not currently found in the manual `hosts` list, and as such
do not get munin monitoring.
2020-07-10 00:14:09 -07:00
Alex Vandiver 24383a5082 puppet: Rename hosts_domain so hosts_prefix can be grepped for. 2020-07-10 00:14:09 -07:00
Alex Vandiver a4e7c7a27e nagios: Remove check_memcached.
check_memcached does not support memcached authentication even in its
latest release (it’s in a TODO item comment, and that’s it), and was
never particularly useful.
2020-07-10 00:12:48 -07:00
Tim Abbott 56addb6139 docs: Fix is_home_view type in API changelog. 2020-07-10 00:08:32 -07:00
Anders Kaseorg ff1622afcf zulip_tools: Replace deprecated mktemp call.
Although mktemp is deprecated due to security issues, this is not a
security issue.

The security problems with mktemp happen when you open the resulting
filename (without O_EXCL) in a publicly writable directory, because
then someone else might have predicted the filename and created or
symlinked or hardlinked something there between the mktemp and the
open, causing you to write to a file you didn’t expect.

Here we don’t open the resulting filename, we symlink to it.  symlink
will refuse to clobber an existing file, and we handle the error that
arises from this case.  This is the normal way to atomically create a
symlink.

We should still replace mktemp because it’s deprecated, but we can’t
replace it with a function that creates the temporary file.  Instead
we build a random filename ourselves.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-09 14:32:02 -07:00
Gittenburg 0b4568d249 message_header: Fix overlap on mobile with flex.
Previously .recipient_row_date was positioned absolutely, allowing it to
overlap with the topic name on narrow screens. This can be solved by
using flexbox. To implement the empty space between the bar controls and
the date on wider screens we move the date outside of the bar controls
(which also makes more sense semantically since the date isn't a
control).

Fixes #15501.
2020-07-09 13:13:38 -07:00
sahil839 67ea4bd937 stream: Fix incorrect dispaly of stream notifications settings.
The stream notification settings checkboxes were not checked
even when the notifications were turned on for the stream.
This was happening because we were passing stream name to
receives_notifications instead of stream id.

This commit fixes the bug by passing stream id to
receives_notifications. This change should have been done
in f3604fb while refactoring receives_notifications to use
stream id instead of name.
2020-07-09 12:46:44 -07:00
Vinit Singh bf31d92fa7 emoji_picker: Fix keys not working if search yields no results.
As a consequence of commit 1113589b9d the backspace key and some other
keys did not respond if the search yielded no results.
This change fixes that bug.
2020-07-09 11:55:13 -07:00
Steve Howell 9662af842f export: Remove stream sanity check.
We also remove the post_process_data option.

The sanity check is just overkill at this point,
since the mechanism to find streams is very
direct due to a recent commit.
2020-07-09 11:34:00 -07:00
Steve Howell 54c596cfc4 export: Just export all streams in a realm.
Before this change we would only export streams
that had actual subscribers, which is usually
harmless, but it was mostly a relic of a one
time migration that we did when we were cleaning
up some dirty data in some of our very early
databases (circa 2016).

Now we work down the table hierarchy in a
more natural way:

    - get Streams in Realm
    - get Recipients matching above Streams
    - get Subscriptions matching above Recipients

Note that for per-user exports, I kept the
same logic (users -> subscriptions -> recipients ->
streams) we had before.

One subtle detail here is that we make our
final Config blocks--which build the final
version of Recipient/Subscription--now hang
off of realm_config.

Fixes #15146.
2020-07-09 11:33:55 -07:00
Aman Agrawal 18bebbf290 recent_topics: Set correct container as scrolling container.
This fixes the bug of extra topics not being rendered on scrolling.
list_render uses `max-height` to determine which container is being
scrolled upon. Set the `max-height` on the scrolling container of
recent topics to help list_render identify it.
2020-07-08 17:51:27 -07:00
Mohit Gupta 79243f7d52 tests: Move test_near_pm_message_url to test_message_fetch.py. 2020-07-08 14:00:05 -07:00
Mohit Gupta 16c3121bc8 tests: Move test_is_private_flag_not_leaked to test_message_flags.py. 2020-07-08 14:00:05 -07:00
Mohit Gupta b0b3cf86ce tests: Move test_auto_subbed_to_personals test to test_signup.py.
This is really a test about account creation.
2020-07-08 13:59:56 -07:00
Mohit Gupta 478956ff0b tests: Move MiscMessageTest to test_audit_log.py. 2020-07-08 13:53:56 -07:00
Mohit Gupta f03c5e35e4 tests: Move NoRecipientIDsTest to test_subs.py. 2020-07-08 13:51:32 -07:00
Vinit Singh 8b54cce833 emoji_picker: Improve emoji picker search results.
This sorts the emoji picker's search results using the same
`typehead.sort_emojis` function as the compose typehead.

Resolves #15694.
2020-07-08 13:47:03 -07:00
Gittenburg 6e5fb85ade refactor: Improve method name in emoji_picker. 2020-07-08 13:25:49 -07:00
Gittenburg 9f20fdc87a refactor: Replace getter with const in emoji_picker. 2020-07-08 13:25:49 -07:00
Gittenburg 08b9aacb88 hotkey: Detect arrow keys while holding shift.
This is useful for the emoji-picker where holding down shift lets you
select multiple emojis. Otherwise when selecting multiple emojis with
the keyboard shortcuts you would need to release shift everytime you
wanted to navigate with the arrow keys.
2020-07-08 13:25:49 -07:00