Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RANGER-4357: Consolidate configs required to enable ORC audit logs #279

Merged
merged 1 commit into from
Aug 30, 2023

Conversation

fateh288
Copy link
Contributor

Other changes: Test cases for orc audits. Typo fix for zlib compression.

What changes were proposed in this pull request?

This is a follow up patch for RANGER-1837 and RANGER-4178 which enables audit logging in ORC format for HDFS audit destination.

The pull request has the following 3 components:

  1. Config consolidation:
    Currently there are 2 configs which will be required to enable orc i.e. xasecure.audit.destination.hdfs.batch.filequeue.filetype=orc and config xasecure.audit.destination.hdfs.filetype=orc
    Ideally, we would want only 1 config to enable orc xasecure.audit.destination.hdfs.batch.filequeue.filetype=orc. Also, orc can only be enabled from filequeue, so above config seems more reasonable.

  2. Fixed a typo for zlib compression

  3. Test cases to verify the ORC audit logs functionality and also verify the above config consolidation works as expected. The test cases also serves as configuration list requirement to enable orc audit logs

How was this patch tested?

Unit tests added to test the ORC feature including config consolidation proposed
Manual testing on cluster also done to verify the functionality

Additional documentation/configuration for reference to enable ORC audit logs (derived from one of the test cases):

xasecure.audit.destination.hdfs.orc.buffersize=100000000000000, 
xasecure.audit.destination.hdfs.batch.filequeue.filespool.dir=target/spool, 
xasecure.audit.destination.hdfs.batch.queuetype=filequeue, 
xasecure.audit.destination.hdfs.batch.filequeue.filespool.buffer.size=100000000000000, 
xasecure.audit.destination.hdfs.batch.filequeue.filetype=orc, 
xasecure.audit.is.enabled=true, 
xasecure.audit.destination.hdfs.filename.format=%app-type%_ranger_audit.orc, 
xasecure.audit.destination.hdfs=enable, 
xasecure.audit.destination.hdfs.orc.stripesize=100000000000000, 
xasecure.audit.destination.hdfs.dir=target/testAuditFileQueueSpoolORC, xasecure.audit.destination.hdfs.orc.compression=snappy, 
xasecure.audit.destination.hdfs.batch.filequeue.filespool.file.rollover.sec=300

…est cases for orc audits. Typo fix for zlib compression.
@rameeshm rameeshm merged commit 5afc852 into apache:master Aug 30, 2023
1 check failed
mneethiraj pushed a commit that referenced this pull request Jun 20, 2024
…est cases for orc audits. Typo fix for zlib compression. (#279)

(cherry picked from commit 5afc852)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants