Commit Graph

1786 Commits

Author SHA1 Message Date
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Tim Abbott 0479843063 docs: Update link to Handlebars Block Helpers. 2020-02-25 11:48:43 -08:00
Steve Howell 59b047c862 docs: Tweak commit message guidelines.
- I fixed a typo with "lowerecase"
- I elaborated on the prefix before elaborating
  on the rest of the message (i.e. went in correct
  order).
- I split out the provision example (since we
  talk about it some depth).
- I added more positive examples.
- I removed the distracting italics around the
  good commit messages.
- I moved the "gather_subscriptions" commit to
  the bottom of the list, since we elaborate
  on that below the list.
2020-02-24 17:56:36 -08:00
Mateusz Mandera 83722275f2 docs: Rename "Testing and writing tests" section to "Testing overview".
The section doesn't really explain anything about actually writing
tests, so "Testing overview" seems like a more fitting name.
2020-02-20 17:21:20 -05:00
Mateusz Mandera 7814f52d45 docs: Replace links to Django 1.11 docs with 2.2 links. 2020-02-19 11:51:18 -08:00
Tim Abbott b1608a51db docs: Update changelog with changes since last release.
This includes an experiment of having a draft of the 2.1.3 changelog,
which is helpful in avoiding duplication with the 2.2.0 changelog for
items we're planning to backport.
2020-02-19 11:50:47 -08:00
Vishnu KS 5bab2a3762 upload: Replace jQuery filedrop with Uppy. 2020-02-13 16:43:19 -08:00
rht 41e3db81be dependencies: Upgrade to Django 2.2.10.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.

Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:

* urls: django.urls.resolvers.RegexURLPattern has been replaced by
  django.urls.resolvers.URLPattern; affects OpenAPI code and related
  features which re-parse Django's internals.
  https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
  ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
  https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
  https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
  the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.

There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.

Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.

Fixes #10835.
2020-02-13 16:27:26 -08:00
Anders Kaseorg d7d8632525 pygments_data: Replace JS module with JSON module.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:09:12 -08:00
Anders Kaseorg e257253e64 emoji_codes: Replace JS module with JSON module.
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.

Update the backend to use emoji_codes.json too instead of the three
separate JSON files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:09:12 -08:00
Chris Bobbe 973c6a3061 docs/contributing/version-control: Link to docs/git/fixing-commits.
For help with using `git rebase -i`.
2020-02-12 09:53:25 -08:00
Tim Abbott 4a36ed6cb2 docs: Advertise support for GitLab authentication. 2020-02-11 14:13:39 -08:00
Dinesh 4304d5f8db auth: Add support for GitLab authentication.
With some tweaks by tabbott to the documentation and comments.

Fixes #13694.
2020-02-11 13:54:17 -08:00
Anders Kaseorg 4c1453df68 code-style: Modernize some ancient JavaScript advice.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-10 14:08:12 -08:00
Anders Kaseorg 4948240619 js: Convert _.filter(a, …) to a.filter(…).
And convert the corresponding function expressions to arrow style
while we’re here.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-10 14:08:12 -08:00
Anders Kaseorg 719546641f js: Convert a.indexOf(…) !== -1 to a.includes(…).
Babel polyfills this for us for Internet Explorer.

import * as babelParser from "recast/parsers/babel";
import * as recast from "recast";
import * as tsParser from "recast/parsers/typescript";
import { builders as b, namedTypes as n } from "ast-types";
import K from "ast-types/gen/kinds";
import fs from "fs";
import path from "path";
import process from "process";

const checkExpression = (node: n.Node): node is K.ExpressionKind =>
  n.Expression.check(node);

for (const file of process.argv.slice(2)) {
  console.log("Parsing", file);
  const ast = recast.parse(fs.readFileSync(file, { encoding: "utf8" }), {
    parser: path.extname(file) === ".ts" ? tsParser : babelParser,
  });
  let changed = false;

  recast.visit(ast, {
    visitBinaryExpression(path) {
      const { operator, left, right } = path.node;
      if (
        n.CallExpression.check(left) &&
        n.MemberExpression.check(left.callee) &&
        !left.callee.computed &&
        n.Identifier.check(left.callee.property) &&
        left.callee.property.name === "indexOf" &&
        left.arguments.length === 1 &&
        checkExpression(left.arguments[0]) &&
        ((["===", "!==", "==", "!=", ">", "<="].includes(operator) &&
          n.UnaryExpression.check(right) &&
          right.operator == "-" &&
          n.Literal.check(right.argument) &&
          right.argument.value === 1) ||
          ([">=", "<"].includes(operator) &&
            n.Literal.check(right) &&
            right.value === 0))
      ) {
        const test = b.callExpression(
          b.memberExpression(left.callee.object, b.identifier("includes")),
          [left.arguments[0]]
        );
        path.replace(
          ["!==", "!=", ">", ">="].includes(operator)
            ? test
            : b.unaryExpression("!", test)
        );
        changed = true;
      }
      this.traverse(path);
    },
  });

  if (changed) {
    console.log("Writing", file);
    fs.writeFileSync(file, recast.print(ast).code, { encoding: "utf8" });
  }
}

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-10 14:08:12 -08:00
Tim Abbott 14e0322dda docs: Fix small issues in architecture overview.
Typos and removing now-inaccurate discussion of the incoming email
integration as a use case for redis.
2020-02-06 15:03:10 -08:00
Mateusz Mandera 00de35b28a docs: Typo fixes in testing/philosophy.md. 2020-02-04 13:58:13 -08:00
Tim Abbott c5af0fccce docs: Fix anchors for using.
We could have just updated the links, but it's probably better the way
it was originally.
2020-02-03 16:55:28 -08:00
Tim Abbott 4e34f672ff docs: Reorganize testing.md and using.md.
This is a fairly involved set of changes, including changes that:

* Delete various legacy or semi-duplicated sections of testing.md.
  Nobody needs to manually delete the postgres datbase anymore, as
  reflected in the fact that the docs still mention postgres 9.1 from
  Ubuntu Precise.
* Simplify the distracting heading section at the top of testing.md.
* Move content on manual testing to docs/development/using.md.
* Moves some content related to managing the database to
  schema-migrations.md. (Resulting in some cleanups to that page as
  well).

I ideally would have split this into smaller pieces.
2020-02-03 16:43:27 -08:00
Tim Abbott 32ff28bf24 docs: Add an article explaining our testing philosophy. 2020-02-03 16:32:35 -08:00
xpac1985 65fe1a9eed
docs: Add info about zulip-announce RSS feed to install docs.
The mailing list can also be subscribed to via RSS/Atom feeds, I just wanted to make that information easier accessible.
2020-01-31 17:24:43 -08:00
Anders Kaseorg 54f8905aa3 release-checklist: Create -dev tags
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-31 16:11:59 -08:00
Rodriq f3cfddce81 docs: Update swagger example to suit V3.0.1. 2020-01-28 14:48:09 -08:00
Tim Abbott d356622594 docs: Add link from LDAP docs to invitation docs.
This addresses confusion we had with some organizations where they
were surprised that with only LDAP enabled, the "invite more users"
feature was available.

Fixes #11685.
2020-01-25 23:41:19 -08:00
Vishnu KS 05b4610381 bots: Remove feedback cross realm bot.
This completes the remaining pieces of removing this missed in
d70e799466 (mostly in tests).
2020-01-25 22:54:44 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
Rodriq 8d466f6a25 docs: Update API docs usage example.
This docs on writing API docs usage example hadn't been updated to use
generate_code_example(curl) after we introduced that feature.
2020-01-22 12:00:10 -08:00
Tim Abbott 445a11753b docs: Update links to Django documentation.
Django 1.9 is no longer on the web.
2020-01-16 16:00:03 -08:00
Tim Abbott c113d74daf docs: Rewrite the guide on using the development environment.
This correct various inaccuracies and adds a bulleted list structure
for better clarity.

I think there's a lot more that could be done here in the form of
linking to other pages, discussing restarting `run-dev.py`, etc.
2020-01-16 15:57:08 -08:00
Chris Bobbe b3901c830b docs/development/using: Add link to ./authentication.
Added a link from docs/development/using ("Using the Development
Environment") to ./authentication ("Authentication in the development
environment") to help people working on the authentication systems
or anyone who needs an API key.
2020-01-16 15:41:12 -08:00
Chris Bobbe 6c3fcc252e docs/development/using: Small wording change.
An attempt to make the point about Django/Tornado server processes
auto-restarting less wordy.
2020-01-16 15:41:12 -08:00
Chris Bobbe 60f3c06d91 docs/development/using: Break into three headings: Server/Web/Mobile.
Separate using.html into Server/Web/Mobile sections so that readers
will find what they're looking for more quickly. Server is at the top
because it contains information relevant to web and mobile developers,
e.g., that the `run-dev.py` console output will provide useful errors.

Fixes #13655.
2020-01-16 15:41:10 -08:00
Tim Abbott c4ba44824d Update changelog for Zulip Server 2.1.2 release. 2020-01-16 12:39:00 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Steve Howell 752d6dc6df tools: Remove find-add-class tool.
I added this tool a few years ago, and I did have
a vision for how it would improve our codebase, but
I can't remember exactly where I was going with it.

At this point the tool is just a little too noisy
to be helpful.  An example of it creating confusion
was a recent PR where somebody was patching
user_circle_class in the PM list, and we already
had similar code in the buddy list, because they
use the same CSS.  I mean, there was possibly a way
that the code could have been structured to remove
some of the duplication, but it probably would have
just moved the complexity around.

I just don't think it's worth maintaining the tool
at this point.
2020-01-14 15:45:49 -08:00
Ray Kraesig 88aa18ba85 docs: link to more-currently-maintained fork of GitX
The well-known rowanj/gitx repository hasn't been updated since 2014.
Preferentially direct new contributors to gitx/gitx instead.

(We retain the rowanj repo as a fallback, since it has precompiled
releases available.)
2020-01-09 16:06:37 -08:00
Steve Howell d258a27a79 refactor: Extract filter_taken_users.
We will take advantage of this in a few more
commits.
2020-01-08 12:55:42 -08:00
Mateusz Mandera e477cae800 docs: Fix missing apostrophe in EMAIL_HOST_USER value. 2020-01-03 16:52:31 -08:00
Mateusz Mandera dc59850d15 docs: Fix incorrect path to get-django-setting script. 2020-01-03 16:52:31 -08:00
Mateusz Mandera d88494deae docs: Add some troubleshooting notes for ldap. 2020-01-03 16:52:30 -08:00
Mateusz Mandera bfb963b9aa docs: Include suggested USERNAME_ATTR in example AD ldap configs. 2020-01-03 16:46:07 -08:00
Tim Abbott 86721c3be5 docs: Fix a few typos in Vagrant docs. 2019-12-30 10:34:48 -08:00
Tim Abbott 18de865daa docs: Simplify a few details in our release checklist. 2019-12-13 17:24:49 -08:00
Tim Abbott b68ff6446c version: Update version and changelog for Zulip 2.1.1 release. 2019-12-13 17:19:45 -08:00
Tim Abbott e38c58e7c7 docs: Rewrite LDAP discussion of AUTH_LDAP_REVERSE_EMAIL_SEARCH.
This moves the mandatory configuration for options A/B/C into a single
bulleted list for each option, rather than split across two steps; I
think the result is significantly more readable.

It also fixes a bug where we suggested setting
AUTH_LDAP_REVERSE_EMAIL_SEARCH = AUTH_LDAP_USER_SEARCH in some cases,
whereas in fact it will never work because the parameters are
`%(email)s`, not `%(user)s`.

Also, now that one needs to set AUTH_LDAP_REVERSE_EMAIL_SEARCH, it
seems worth adding values for that to the Active Directory
instructions.  Thanks to @alfonsrv for the suggestion.
2019-12-13 13:55:52 -08:00
Vishnu KS 6901087246 install: Use crudini for storing value of POSTGRES_MISSING_DICTIONARIES.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.

Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
2019-12-13 12:05:39 -08:00
Rohitt Vashishtha dc4181beec minor: Fix typo in changelog. 2019-12-12 22:52:09 -08:00
Tim Abbott 03a3ae8b61 Release Zulip Server 2.1.0. 2019-12-12 22:23:22 -08:00
Tim Abbott 35959d43c4 docs: Clean up troubleshooting guide.
This article is definitely still below our polish goals, but this is
also definitely an improvement.
2019-12-12 22:19:12 -08:00