REASSESSING FOUR KEY CHALLENGES IN BIG DATA RESEARCH: PRIVACY, DATA QUALITY, INFRASTRUCTURE, AND UNCERTAINTY

Author

Dr. A. Antony Prakash , S. Keerthana

Abstract

 Large-scale, continuously generated data has moved from a technical curiosity to a routine input for decision-making in business, government, and science. Realising the value of this data, however, keeps running into the same four obstacles: keeping individuals' information private, trusting the quality of what has been collected, building infrastructure that can actually store and process it, and coping with the fact that much of it arrives incomplete. This review reads four recent contributions against one another to see how far each has moved the field on its own obstacle, and where the four still fail to connect. One traces privacy research from k-anonymity through differential privacy and across disciplines as different as cryptography and public policy. A second consolidates three decades of information-systems scholarship on what "good" data actually means and how that quality feeds into decision performance. A third maps the storage, computation, and visualisation problems that big data creates, alongside the tool ecosystem — Hadoop, Spark, Storm, and related platforms — built to solve them. A fourth proposes a deep-learning method for classifying data in which entire modalities are missing, treating that missingness as something to be modelled rather than patched over. Read together, the four works share a recurring trade-off between usefulness and protection, a shift away from deterministic fixes toward probabilistic ones, and a persistent gap between elegant theory and systems that work at genuine scale. The review closes by naming where these strands could be combined and where, at present, they are not.

Keywords

big data; data privacy; differential privacy; data quality; analytics infrastructure; data analytics competency; incomplete multi-view learning; uncertainty estimation.

Full Text:

Download Paper PDF

References

[1]Acharjya, D. P., & Ahmed, K. P. (2016). A survey on big data analytics: Challenges, open research issues and tools. International Journal of Advanced Computer Science and Applications, 7(2), 511-518.
[2]Bharadwaj, A. S. (2000). A resource-based perspective on information technology capability and firm performance: An empirical investigation. MIS Quarterly, 24(1), 169-196.
[3]De Montjoye, Y.-A., Radaelli, L., Singh, V. K., & Pentland, A. (2015). Unique in the shopping mall: On the reidentifiability of credit card metadata. Science, 347(6221), 536-539.
[4]Dwork, C. (2006). Differential privacy. In Proceedings of the 33rd International Colloquium on Automata, Languages and Programming (ICALP) (pp. 1-12).
[5]Huber, G. P. (1990). A theory of the effects of advanced information technologies on organizational design, intelligence, and decision making. Academy of Management Review, 15(1), 47-71.
[6]Li, N., Li, T., & Venkatasubramanian, S. (2010). Closeness: A new privacy measure for data publishing. IEEE Transactions on Knowledge and Data Engineering, 22(7), 943-956.
[7]Machanavajjhala, A., Kifer, D., Gehrke, J., & Venkitasubramaniam, M. (2007). ℓ-diversity: Privacy beyond k-anonymity. ACM Transactions on Knowledge Discovery from Data, 1(1), Article 3.
[8]Samarati, P., & Sweeney, L. (1998). Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression. In Proceedings of the IEEE Symposium on Research in Security and Privacy (pp. 1-19).
[9]Wand, Y., & Wang, R. Y. (1996). Anchoring data quality dimensions in ontological foundations. Communications of the ACM, 39(11), 86-95.
[10]Wang, J., Liu, Y., Li, P., Lin, Z., Sindakis, S., & Aggarwal, S. (2023). Overview of data quality: Examining the dimensions, antecedents, and impacts of data quality. Journal of the Knowledge Economy, 15, 1159-1178. https://doi.org/10.1007/s13132-022-01096-6
[11]Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5-33.
[12]Xie, M., Han, Z., Zhang, C., Bai, Y., & Hu, Q. (2023). Exploring and exploiting uncertainty for incomplete multi-view classification. arXiv preprint arXiv:2304.05165.
[13]Yu, S. (2016). Big privacy: Challenges and opportunities of privacy study in the age of big data. IEEE Access, 4, 2751-2763. https://doi.org/10.1109/ACCESS.2016.2577036

Share your valuable work from Social Media Buttons