Commit Graph

184 Commits

Author SHA1 Message Date
rht 33b1a541d7 zerver/lib: Use python 3 syntax for typing.
With tweaks by tabbott to fix line spacing.
2017-11-18 16:09:04 -08:00
rht 09af29b051 zerver/lib: Text-wrap long lines exceeding 110. 2017-11-15 10:58:03 -08:00
rht cceba2023c Use makedirs directly instead of wrapping it in mkdirs. 2017-11-14 09:47:09 -08:00
rht 72ce1c6501 Use a more concise `mkdir -p` when the situation permits. 2017-11-14 09:47:09 -08:00
Steve Howell 3158ed01bf Remove force_str() in upload.py. 2017-11-09 10:43:20 -08:00
Steve Howell b0bb09cdb8 Remove force_text() in get_signed_upload_url(). 2017-11-09 10:43:20 -08:00
Steve Howell 7da6beac99 Remove force_text() in upload.get_file_info(). 2017-11-09 10:43:20 -08:00
Steve Howell a9f42e6266 Remove force_text() from sanitize_name(). 2017-11-09 10:43:20 -08:00
rht 5ee40bf718 Remove usage of six.moves.binary_type. 2017-11-09 10:00:00 -08:00
rht 19bd335cbb Change urllib import to be Python 3-specific. 2017-11-07 10:46:42 -08:00
rht e311842a1b zerver/lib: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
rht fef7d6ba09 zerver/lib: Remove u prefix from strings.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-03 15:34:37 -07:00
rht f43e54d352 zerver/lib: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Tim Abbott 5184c64a92 upload: Fix uploading files with Python 3.
Apparently there's a bug in typeshed's stubs for uploading files,
which resulted in our S3 upload code being broken on Python 3.
2017-08-16 07:33:23 -07:00
Harshit Bansal 5a5072730c realm_emoji: Don't accept animated realm emoji with unequal dimensions.
Fixes: #3654.
2017-08-05 18:07:21 -07:00
Greg Price 37dbcefa58 upload: Use the new ErrorCode system in our custom error classes.
All JsonableError subclasses now have corresponding ErrorCode values
of their own, reducing the number of different patterns for using
the new JsonableError API.
2017-07-24 16:41:22 -07:00
Nathan Miller 2311e169ec mypy: Various strict-optional fixes in zerver. 2017-05-26 10:10:20 -07:00
Rick Chern 70d68f7e71 Refactoring: Replace get_user_profile_by_email() in lib/upload.py 2017-05-23 12:37:49 -07:00
Aditya Bansal 85aa07e2d4 pep8: Add compliance with rule E261 to zerver/lib/upload.py. 2017-05-18 03:00:32 +05:30
K.Kanakhin f13d6a18eb realm-emoji: Add realm emoji uploading instead url providing.
- Add file_name field to `RealmEmoji` model and migration.
- Add emoji upload supporting to Upload backends.
- Add uploaded file processing to emoji views.
- Use emoji source url as based for display url.
- Change emoji form for image uploading.
- Fix back-end tests.
- Fix front-end tests.
- Add tests for emoji uploading.

Fixes #1134
2017-05-01 14:50:20 -07:00
Tim Abbott e90748348b attachment: Remove unused claim_attachment return value. 2017-04-18 21:27:29 -07:00
Tim Abbott 568b59291b attachment: Improve rules for managing attachment ownership.
The previous logic was that anyone with a link to a file could send it
to other users, but only the owner could make a file realm-public.
This had some confusing corner cases.

The new logic is much simpler:
* Only the file's owner/uploader can include a file in a message for
  the first time.
* Anyone with access to read a file can share it with others by
  including it in messages they send.
* Once a file has been sent to a public stream, any user in the realm
  can access it.
2017-04-18 21:27:29 -07:00
Tim Abbott 1747427bf3 upload: Fix URL for realm icons with S3 backend. 2017-04-12 19:48:24 -07:00
Tim Abbott 8775a22663 avatar: Fix buggy avatar URL scheme with S3 backend.
Also adds tests for the avatar URL scheme.
2017-03-21 16:12:04 -07:00
adnrs96 3f58e20993 Refactor: Change user_avatar_hash with user_avatar_path at all calls.
In this commit we change user_avatar_hash with user_avatar_path which
now returns paths to avatars based on the email hash.

Tweaked by tabbott to avoid an import loop.
2017-03-04 18:39:49 -08:00
Philip Skomorokhov 866a7b06b2 upload: Limit total size of files uploaded by a user to 1GB.
Fixes #3884.
2017-03-04 18:08:30 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
adnrs96 9eb47f108c Refactor: Change upload_avatar_image to accept two user profiles.
In this commit we just change the upload_avatar_image function to accept
two user_profiles acting_user_profile and target_user_profile. Basically
email param is dropped for a target_user_profile so that avatar's could
be moved lateron to user id based storage.
2017-03-03 18:15:15 -08:00
PhilSk 53f3d84af2 attachment: Add 'size' field tracking size of uploaded files.
This tracking will make it possible in the future to limit the total
size of uploads on a per-user or per-organization basis.

Fixes #3774.
2017-03-01 15:58:21 -08:00
K.Kanakhin 257bb40698 realm-icon: Add realm icon feature.
- Add realm icon fields to realm model.
- Add migration for new realm model's field.
- Add views for icon uploading and deleting.
- Add routes for realm icons views.
- Add JS widget for realm icon upload setting.
- Add realm icon upload to administration
  organization setting.
- Add tests for realm icons.

Fixes #3660.
2017-02-26 12:16:07 -08:00
Tim Abbott c5ed119f89 upload: Rename BadImageError exception to not mention avatars.
This is preparation for using that function to decode realm icons as
well.
2017-02-26 12:16:07 -08:00
Tim Abbott 9f0c1db430 mypy: clean up strict optional errors in upload.py. 2017-02-10 23:53:44 -08:00
Tim Abbott bb5d81281c mypy: Upgrade to new package name and version 0.571.
Fixes #3448.
2017-02-10 23:53:44 -08:00
Eklavya Sharma dd0e1f6a4c Use correct string type in boto function parameters.
boto's stubs have been updated in mypy 0.4.7, which has given us
more information about what type of strings are expected as
parameters in various functions.
2017-02-06 22:37:37 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott 3f8d4193da lint: Fix % comprehensions being used without a tuple. 2017-01-09 11:45:11 -08:00
Rishi Gupta cf762eaf84 Change X.realm.id to X.realm_id across codebase.
This makes it more clearly the pattern in the Zulip codebase, and thus
decreases the risk of accidentally doing database queries.
2017-01-03 16:46:26 -08:00
Robert Hönig 0917493588 mypy: Convert zerver/lib to use typing.Text. 2016-12-25 10:33:45 -08:00
Bickio 6b0df43463 pep8: Fix E125. 2016-11-30 20:03:29 -08:00
Brock Whittaker fca61b2031 Add medium size avatars for use on the user's own settings page.
This adds a medium (500px) size avatar thumbnail, that can be
referenced as `{name}-medium.png`.  It is intended to be used on the
user's own settings page, though we may come up with other use cases
for high-resolution avatars in the future.

This will automatically generate and upload the medium avatar images
when a new avatar original is uploaded, and contains a migration
(contributed by Kirill Kanakhin) to ensure all pre-existing avatar
images have a medium avatar.

Note that this implementation does not provide an endpoint for
fetching the medium-size avatar for another user.

[substantially modified by tabbott]
2016-10-25 09:42:14 -07:00
Tim Abbott ac2007dd9b resize_avatar: Add a size option. 2016-10-02 21:29:04 -07:00
Tim Abbott 87b9017845 upload: Extract get_avatar_url and add to UploadBackend. 2016-10-02 21:20:43 -07:00
Tim Abbott 22fd7ba02a avatar: Move avatar hash computations to their own file. 2016-10-02 21:19:10 -07:00
Umair Khan fd9dd51d16 Url encoded name of the file should be an ascii.
The url encoded name of the file should not be a unicode. This
results in an error when we later try to unquote it.

Fixes: #1803
2016-09-26 21:13:34 -07:00
rahuldeve c61d149837 Pass Message object instead of message dict to claim_attachment. 2016-07-25 17:51:17 -07:00
Tim Abbott 34a251adb1 upload: Fix exception uploaded files with unknown content type.
It turns out our detected content_type can actually be None, which
meant calling force_text on it broke uploading some files.
2016-07-13 10:57:55 -07:00
Tim Abbott e21bc11cfd Extract attachment_url_to_path_id. 2016-07-10 18:07:37 -07:00
Tim Abbott 48ae178d0b Refactor attachment_url_re to be a common value in upload.py. 2016-07-10 18:04:58 -07:00
Eklavya Sharma b76dc9bf4e zerver/lib/upload.py: Fix string types. 2016-06-29 20:43:28 +05:30
rahuldeve 8cecb37743 Modify Attachment model to track file access permissions. 2016-06-23 17:46:16 -07:00
Umair Khan a976ccefbf [third] Urlencode name of the uploaded file.
Update jquery-filedrop to send urlencoded filenames.

As discussed in https://github.com/zulip/zulip/pull/1023, this fix is
already in jquery-filedrop upstream.

Fixes #981.
2016-06-14 15:32:41 -07:00
Eklavya Sharma 7ec9cb7e93 Annotate model fields: Attachment.
Also fix clashing annotations.
2016-06-13 20:01:03 +05:30
Eklavya Sharma 71e613424b Fix annotations clashing with UserProfile's model fields. 2016-06-13 20:01:01 +05:30
Eklavya Sharma 2841aa642d Fix annotations related to make_safe_digest and hashes. 2016-06-12 09:30:53 -07:00
rahuldeve fa13582ffb Serve uploaded files through get_uploaded_file in development.
Previously, uploaded files were served:
* With S3UploadBackend, via get_uploaded_file (redirects to S3)
* With LocalUploadBackend in production, via nginx directly
* With LocalUploadBackend in development, via Django's static file server

This changes that last case to use get_uploaded_file in development,
which is a key step towards being able to do proper access control
authorization.

Does not affect production.
2016-06-09 22:58:25 -07:00
Tim Abbott 719e5487b9 upload: Use classes to define S3/Local upload backends.
This has no functional changes; we just replace the old hacky
assignment of functions with assignment of the upload backend to a
variable.

I'm not totally happy with this, because we end up having to copy the
type annotations of the three methods 4 times each, but this should
make it a lot easier to test the (non-default-in-tests) S3 backend
using end-to-end tests, which would have caught
13bac1cc2a.

I expect we'll iterate on the interface over time; ideally, I'd like
all the code that checks LOCAL_UPLOADS_DIR to be inside upload.py, and
primarily in these classes.
2016-06-09 22:45:03 -07:00
rahuldeve 3e3462da0d Refactor zerver.lib.upload.upload_message_image_through_web_client.
upload_message_image_through_web_client -> upload_message_image_from_request
2016-06-09 21:09:12 -07:00
rahuldeve 23ff717bee Fix annotations: zerver.lib.upload. 2016-06-08 09:49:14 -07:00
medullaskyline 47c3ec1283 Annotate zerver.lib.upload. 2016-06-04 19:13:03 -07:00
Umair Khan 08fbd57245 [i18n] Make error messages translatable.
Make all strings passing through `json_error` and `JsonableError`
translatable.

Fixes #727
2016-05-31 07:40:42 -07:00
Tim Abbott 572c69f3c2 Move REQ and friends to their own module. 2016-05-30 11:24:17 -07:00
Umair Khan 5359e6b0d4 Convert Zulip to use Jinja2 templates.
This results in a substantial performance improvement for all of
Zulip's backend templates.

Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.

Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`

Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
  instead of `RequestContext` object. There are two reasons to do this:
    1. `RequestContext` is not compatible with Jinja2
    2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
  includes a custom backend and a template renderer, compressors for js
  and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.

Fixes #620.
2016-05-09 09:55:18 -07:00
rahuldeve dde832b158 Add Attachment model to keep track of uploads.
This commit adds the capability to keep track and remove uploaded
files.  Unclaimed attachments are files that have been uploaded to the
server but are not referred in any messages.  A management command to
remove old unclaimed files after a week is also included.

Tests for getting the file referred in messages are also included.
2016-05-02 22:14:47 -07:00
Eklavya Sharma 4e18d856e3 Prevent 500 error when user uploads invalid avatar.
When uploaded avatar image is not a valid image file, PIL raises
IOError. Catch the IOError raised by PIL and raise JsonableError.
This will return a response with status code 400.
2016-04-20 12:00:13 +05:30
Varshit e0ef1a991e Rewrite sanitize_name to better preserve filenames.
The previous version of sanitize_name dropped all unicode characters
and mangled filenames with multiple `.`s in the extension, leading to
confusing URLs for files uploaded to Zulip.

Fixes #321.

[tweaked significantly by tabbott]
2016-04-03 16:26:12 -07:00
Tim Abbott 757e89260e Migrate use of StringIO to Python 2+3 compatible six.moves.cStringIO.
And add a check for additional usage of the original StringIO module.
2016-01-26 21:09:43 -08:00
Luke Faraone 8f8b2519ea Redirect legacy URLs to their new secure location.
URLs with a realm of "unk" will be queried against the new bucket to
determine the relevant realm of the uploading user.

(imported from commit 5d39801951face3cc33c46a61246ba434862a808)
2014-05-05 20:26:29 -07:00
Tim Abbott 8b74a3e052 Remove unauthenticated file upload support from Zulip.
(imported from commit 97262590ac5ad56c18f415fa1c777510aed2baeb)
2014-05-05 16:14:09 -07:00
Waseem Daher ef23a4efe4 Disable authed uploads for zulip.com.
(imported from commit 3c432e730005e4c8404245c974b53c5a49fd4c9d)
2014-03-11 13:06:19 -04:00
Jessica McKellar 25a9eae74b Enabled authed uploads for Bargain Bit.
(imported from commit f793b7ac41341384b914168a3fc5ed637d63a645)
2014-03-05 11:55:02 -05:00
Luke Faraone fe55127eee Override realm of uploading user when invoked via the email mirror
Here, we don't want to check the uploading users' realm when determining
message privacy, because that'll prevent non-Zulip users from having
email-mirror-uploaded images. Instead, we just pass along the target
realm for the message explicitly to upload_message_image()

(imported from commit 6891261552135b1f41ff9da55ffe963ee5000556)
2014-02-07 09:45:57 -08:00
Luke Faraone 1e87e7c5c1 Switch authed_upload_enabled to check for a realm, instead of a UP
This is always a realm setting anyway, and it'll make reuse easier.

(imported from commit c64e1901b60051608cee3cf9a6b3aef8630ae237)
2014-02-07 09:45:57 -08:00
Steve Howell da858afacf Save on S3 bills by passing validate=False into conn.get_bucket().
(See the comments in the code for details.)

(imported from commit e518b9d89aca8a6358c826bf7e7f03381e0560f4)
2014-02-06 12:14:14 -05:00
Tim Abbott 68dcc760c3 Clean up some unused imports.
(imported from commit 0c5d8e2a55ba1b8909ba807fee3afe863dcdc226)
2013-11-04 11:51:17 -05:00
Kevin Mehall 06590f4332 Upload to local filesystem on local server.
(imported from commit 6c38a8cae721b2e7a0863470692cc56425006ecb)
2013-10-29 16:19:04 -04:00
Kevin Mehall f9af7ad1cf Resize uploaded avatar images
(imported from commit 47ff86fcd47871916631baa34f99bbddb67f87d0)
2013-10-29 11:42:33 -04:00
Kevin Mehall 19b9b6e2ec Reduce the length of public upload URLs.
(imported from commit dc4d51da39b218068012ebad57e1f92cb50137c3)
2013-10-24 17:01:06 -04:00
Kevin Mehall 0a271f149b Enable authed file uploads for squarespace
(imported from commit c72e76d27f99cd7610eb6c93b57b8ff727b461a9)
2013-10-24 17:01:06 -04:00
Kevin Mehall 0a3a22cb3d Support authenticated upload URLs.
Trac #1734

This is implemented by bouncing uploaded file links through a view
that checks authentication and redirects to an expiring S3 URL.

This makes file uploads return a domain-relative URI. The client converts
this to an absolute URI when it's in the composebox, then back to relative
when it's submitted to the server.

We need the relative URI because the same message may be viewed across
{staging,www,zephyr}.zulip.com, which have different cookies.

(imported from commit 33acb2abaa3002325f389d5198fb20ee1b30f5fa)
2013-10-24 17:01:06 -04:00
Kevin Mehall 473a98d0a4 Split out common code for existing and authenticated upload systems.
(imported from commit bf513eaff90ae7c4a0be825317690058f9f54621)
2013-10-24 17:01:06 -04:00
Kevin Mehall fb4eae9e2b Include realm ID in S3 metadata on uploaded files.
(imported from commit 7d41ec94991bd257485116791a0d1ef2c22d9b37)
2013-10-24 17:01:05 -04:00
Jessica McKellar 97e6709d49 uploads: have upload_image_to_s3 take file data instead of a file object.
We have to read the data anyway, and we don't have a convenient file
handle for uploads from attachments sent through the e-mail gateway.

(imported from commit 86260a4eaceef85c82707929a80558e11dc54ef6)
2013-09-17 10:00:01 -04:00
Jessica McKellar 8a40fb346c Be able to upload images to S3 from outside the web client.
We'll need this to upload attachments sent through the email gateway.

(imported from commit b2f5eb736b65f1478189c3aebd4cb37f1ad540fb)
2013-09-17 10:00:01 -04:00
Tim Abbott e111a2f9a5 [manual] Rename Django app from zephyr to zerver.
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).

At the time it is deployed, we need to make a few changes directly in
the database:

(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';

(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
2013-08-06 07:39:36 -04:00